Paper deep dive
Barnamala: Parameter-Efficient Handwritten Devanagari Recognition at Benchmark Saturation
Ashish Thapa, Samrat Karki
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 7/16/2026, 8:18:20 AM
Summary
The paper introduces Barnamala, a highly compact (1.11M parameters) convolutional network for 46-class handwritten Devanagari recognition. It achieves state-of-the-art accuracy (99.73%) on the DHCD benchmark, matching larger models like MallaNet (17.32M parameters) while being 15.6x smaller and 9.5x faster. The authors demonstrate that the DHCD benchmark has reached saturation, with an intrinsic error floor of 11 misclassifications that no model configuration can overcome. Barnamala also shows superior transfer learning capabilities to other datasets (CMATERdb, NHCD) and significantly better robustness to image corruptions compared to larger baselines.
Entities (9)
Relation Signals (8)
Barnamala → hasparametercount → 1.11M
confidence 99% · compact convolutional network (1.11 M parameters)
Barnamala → achievesaccuracyon → DHCD
confidence 98% · reached 99.73%, the highest reported on the standard 46-class split
DHCD → hasintrinsicerrorfloor → 11 errors
confidence 97% · every model tested, large teacher ensembles included, hits the same 11-error intrinsic floor.
MallaNet → hasparametercount → 17.32M
confidence 97% · MallaNet [9] reaches 99.71 % with a 17.32 M-parameter convolutional architecture
Barnamala → outperformsinrobustness → MallaNet
confidence 96% · corruption robustness is also far better than large baselines (mean corruption accuracy 75.7% vs. 38.7%).
Barnamala → usestechnique → Knowledge Distillation
confidence 96% · We follow the knowledge distillation framework of Hinton et al. [5].
Barnamala → transfersto → CMATERdb
confidence 95% · Outside of DHCD, zero-shot on CMATERdb digits gives 76.6% and fine-tuning reaches 97.8%
McNemar Test → usedforcomparison → Barnamala
confidence 94% · The exact McNemar test across all student-teacher pairs answers directly...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We built a compact convolutional network (1.11 M parameters) for 46-class DHCD Devanagari recognition and reached 99.73%, the highest reported at 15.6x smaller than prior state-of-the-art. We have effectively reached the saturation point: every model tested, large teacher ensembles included, hits the same 11-error intrinsic floor. No configuration achieves a statistically clear win under exact McNemar tests with Wilson confidence intervals. Even without knowledge distillation, our student matches the nearest large-model baseline (17.32 M parameters; McNemar $p = 0.345$). Outside of DHCD, zero-shot on CMATERdb digits gives 76.6% and fine-tuning reaches 97.8%; corruption robustness is also far better than large baselines (mean corruption accuracy 75.7% vs. 38.7%). All artifacts are at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2607.13689v1
- Canonical: https://arxiv.org/abs/2607.13689v1
Trouble viewing inline? Open PDF directly →
Full Text
33,872 characters extracted from source content.
Expand or collapse full text
Barnamala: Parameter-Efficient Handwritten Devanagari Recognition at Benchmark Saturation Ashish Thapa 1 and Samrat Karki 2 1 Ampixa Labs ashish@ampixa.com 2 Pulchowk Campus 082bel077.samrat@pcampus.edu.np Abstract. We built a compact convolutional network (1.11 M parame- ters) for 46-class DHCD Devanagari recognition and reached 99.73 % [1], the highest reported at 15.6× smaller than prior state-of-the-art. We have effectively reached the saturation point: every model tested, large teacher ensembles included, hits the same 11-error intrinsic floor. No configuration achieves a statistically clear win under exact McNemar tests with Wilson confidence intervals. Even without knowledge distil- lation, our student matches the nearest large-model baseline (17.32 M parameters; McNemar p = 0.345). Outside of DHCD, zero-shot on CMA- TERdb digits gives 76.6 % and fine-tuning reaches 97.8 %; corruption robustness is also far better than large baselines (mean corruption ac- curacy 75.7 % vs. 38.7 %). All artifacts are at https://github.com/ Ampixa/barnamala. 1 Introduction The DHCD dataset [1] is the standard benchmark for 46-class handwritten De- vanagari character recognition, and a widely used reference point for Indic script recognition more broadly; its samples were collected from school students. Top-1 accuracies have crowded into a narrow band above 99.7 %: MallaNet [9] reaches 99.71 % with a 17.32 M-parameter convolutional architecture; Mishra et al. [11] independently report 99.72 % with a similarly large model. At this level suc- cessive gains are measured in hundredths of a percent, yet models still carry parameter budgets far beyond what an on-device recognizer can sustain. Existing DHCD papers have not applied the scrutiny that 99%-range results demand. Point accuracy dominates; when McNemar testing appears at all, it is applied selectively against weaker baselines without exact p-values. Especially near 99.7 %, with n=13,800 test samples, raw accuracy rankings carry little inferential weight. The decisive issue is paired. Do the two models fail on different examples? The exact McNemar test across all student-teacher pairs answers directly, with Wilson confidence interval bounding the accuracy of each model. Once this dis- cipline is applied, the leaderboard will look different. arXiv:2607.13689v1 [cs.CV] 15 Jul 2026 2A. Thapa and S. Karki A 1.11 M-parameter student reaches statistical parity with the prior state- of-the-art [9] (McNemar p = 0.345). That holds even without knowledge distilla- tion. Neither the distillation target (TTA-softened versus clean) nor the ensemble size (9 versus 15 teachers) makes a statistically significant difference. Every con- figuration tested converges to the same 11-error intrinsic floor; progress now requires a harder evaluation set or substantially different modelling assumptions. Beyond accuracy numbers, a compact convolution network trained on one recognition task transfers. The pipeline can be reproduced for other character recognition problems. Contributions. C1. Barnamala—1.11 M parameters—achieves 99.73 % on DHCD, the highest reported on the standard 46-class split, at 15.6× smaller footprint than prior state-of-the-art. McNemar (p = 0.345) confirms parity even without distillation. C2. DHCD is saturated. Exact McNemar tests and Wilson CIs applied across all configurations reveal a shared 11-error intrinsic floor that no config- uration escapes. C3. The ablations settle which choices matter: distillation target cleanliness, en- semble size (9 vs. 15 teachers), and TTA each produce no statistically sig- nificant gain once noise is properly accounted for. C4. At 15.6× smaller and 9.5× faster on CPU than prior state-of-the-art, Bar- namala transfers to CMATERdb digits (zero-shot 76.6%; fine-tuned 97.8%) and degrades far less under corruptions (mCA 75.7% vs. 38.7%). C5. Full reproduction materials are at https://github.com/Ampixa/barnamala. 2 Related Work Indic and Devanagari handwritten character recognition. The DHCD dataset [1] has been the standard 46-class benchmark since 2015. MallaNet [9] couples residual branch merging with homogeneous filter capsule (HFC) layers to reach 99.71 % at 17.32 M parameters; Mishra et al. [11] independently report 99.72 % with a comparably large architecture; for practical purposes, indistinguishable. A concurrent preprint puts 99.80 % on the 10-class digit subset [8]; digit-only results do not transfer to the 46-class task. Knowledge distillation and compact model design. Hinton et al. [5] showed that soft probability outputs (“dark knowledge”) transfer accuracy from a large teacher to a smaller student at substantially lower capacity. He et al. [4] introduced residual identity mappings; Hu et al. [6] added squeeze-and-excitation channel re-weighting. Both are now routine components in compact architectures that trade parameter count for retained discriminative power. No prior work has applied knowledge distillation to the 46-class DHCD task. Stress-testing distilla- tion on a saturated benchmark, where the performance gap sits inside sampling variation, is likewise unexplored. Barnamala: Parameter-Efficient Devanagari Recognition3 Input 1×32×32 Stem Conv 3×3, s1 BN→ReLU 40×32×32 Stage 1 2×PreAct SE no stride 40×32×32 39 k params Stage 2 2×PreAct SE stride 2 80×16×16 154 k params Stage 3 2×PreAct SE stride 2 160×8×8 616 k params BN→ReLU Global AvgPool 160-dim Dropoutp=0.1 FC 160→46 46 classes (a) Full pipeline in C×H×W BNReLU Conv 3×3 BNReLU Conv 3×3 ⊗⊕ out C×H×W pre-activation: BN before Conv improves gradient flow 3×3kernel: captures local stroke patterns ⊗: channel-wise scale GAP FCC/8 ReLU FCC σ SE:recalibrate channel weights (r=8) identity (or 1×1 Convs=2 when downsampling) (b) PreAct SE-ResBlock (unravelled) Fig. 1: Barnamala architecture. (a) Full pipeline: a stem convolution feeds three stages of PreAct SE-ResBlocks with channel widths (40, 80, 160); a BN–ReLU– GAP head produces a 160-dim descriptor before the linear classifier. (b) Unrav- elled PreAct SE-ResBlock: the main path runs two rounds of BN, ReLU, and Conv 3×3. The SE module then squeezes channel weights with a GAP followed by two FCs (ratio r=8) and a sigmoid; the re-weighted features sum with an identity skip (or 1×1 Conv when downsampling) at ⊕. Statistical significance and benchmark saturation. Raw accuracy rankings on DHCD can flip between runs with no change in model quality, once the best models err on fewer than 0.3 % of examples. McNemar’s test [10] handles this: by conditioning on disagreement examples, it detects genuine error-pattern shifts without inflating sensitivity from the large pool of shared correct predictions. Wilson score intervals [14] bound single-model accuracy; [3] links confidence to empirical accuracy, which matters when soft distillation targets are being inter- preted. [9] applies McNemar against four weaker baselines but omits its nearest competitor (Mishra et al., 99.72 %) and reports no exact p-values. Barnamala closes this gap by reporting exact McNemar p-values and Wilson CIs across every comparison and showing the benchmark is saturated. 4A. Thapa and S. Karki 3 Method 3.1 Compact Student Architecture Barnamala is a pre-activation squeeze-and-excitation residual network [4,6] de- signed for single-channel 32×32 images. Channel widths run (40, 80, 160) across three stages of depth (2, 2, 2); Stages 2 and 3 downsample by stride 2 (Figure 1). The total trainable parameter count is 1,109,116. 3.2 Teacher Ensemble The ensemble consists of 15 teachers drawn from three architectural configu- rations intersected with five independent random seeds (seeds 0 to 4). Two of those configurations share widths (96, 192, 384) and depths (3, 3, 3) but differ in augmentation intensity (medium vs. heavy; see Section 3.4); the third uses widths (64, 128, 256) with depths (4, 4, 4) under medium augmentation. The re- sulting parameter counts are 5.97 M and 9.89 M per teacher. The spread across configurations is intentional to discourage mode collapse in the soft targets. All teachers share the same Barnamala codebase (DevNet class) and are trained without distillation. Their logits are dumped once into a single NumPy archive reused across all student runs. 3.3 Ensemble Knowledge Distillation We follow the knowledge distillation framework of Hinton et al. [5]. The ensemble target for each training image is the mean softmax of nine teacher checkpoints (configs A, B, C × seeds 0–2), where each teacher’s softmax is computed with horizontal-flip test-time averaging over the training image. The distillation ob- jective combines a temperature-softened KL divergence and a hard cross-entropy term: L = α· T 2 KL σ z t T σ z s T + (1− α)· CE(z s ,y),(1) where z s and z t are student and (mean) teacher logits, σ denotes the softmax, T = 4.0 is the temperature, and α = 0.7 weights the soft target. Label smoothing (ε = 0.1) is applied to the hard cross-entropy term. KL is computed in the teacher-to-student direction. The choice of target-generation scheme (clean no- TTA vs. flip-averaged) and ensemble size (9 vs. 15 teachers) are examined in Section 4.5. 3.4 Script-Aware Training Devanagari Script is not horizontally symmetric. Reflecting glyphs produces in- valid characters. Horizontal flips are completely excluded. Instead, the training transformation applies affine perturbations (rotation, translation, scale, shear) followed by elastic deformations. The strength is controlled by an augmenta- tion tier (light/medium/heavy) selected per configuration. Random erasing (p = 0.25, area 2–10%) rounds out the per-image regularisation. Barnamala: Parameter-Efficient Devanagari Recognition5 The mixing extensions, mixup [17] (α = 0.2) and CutMix [16] (α = 1.0), are selected with equal probability and each is activated for each batch with an over- all probability of 0.5. Both are turned off while distillation is running. The KD loss (formula 1) always shows a clean, unenhanced teacher prediction, because the per-image teacher logits cannot be linearly combined without compromising the soft target distribution. We optimise with AdamW using a cosine learning-rate schedule preceded by a 5-epoch linear warm-up. A weight exponential moving average (EMA, decay 0.999) tracks a shadow copy; validation and the final checkpoint use the EMA weights. 3.5 Evaluation Protocol Reported accuracies come from a single forward pass over the DHCD held-out test split; no test-time augmentation is applied. Model selection relies on a 10 % stratified validation set carved from the training split before augmentation. For statistical comparisons between Barnamala and the baseline [9], we use the exact two-sided McNemar test [10] on per-sample binary correctness indica- tors. Confidence intervals for accuracy and error rate are Wilson score intervals at 95% [14]. ECE [3] is computed using 15 equal width bins on the held-out test set. 4 Experiments 4.1 Experimental Setup Dataset. We use DHCD [1], the standard 46-class Devanagari handwriting bench- mark (36 consonants, 10 digits): 78,200 training images and 13,800 test images, each 32×32 grayscale. Reproducing the baseline. To enable sample-level McNemar comparisons, we reimplemented the architecture of [9] from the published description, including exact normalization (mean=0.5, std=0.5) and the class-index permutation im- plied by the data loader. Seed 0 produces 40 errors (99.7101%), which matches the reported number. 4.2 Main Parity Result Five seeds per configuration; Table 1. A 1.11 M-parameter student sits inside the noise floor of a 17.32 M-parameter baseline. The distilled, TTA-targets variant (9 teacher checkpoints, flip-averaged soft labels; see §3.3) reaches 36.6±2.1 errors (99.735%), with McNemar p = 0.345: no detectable gap at α = 0.05. Best seed: 34 errors (99.754%), Wilson 95% CI [99.656%, 99.824%], at 15.6× fewer parameters. Clean single-pass logits from a 15-teacher pool give p = 1.000; the supervised control, trained with no teacher signal, ties at p = 0.701. Distillation is not the source of parity. 6A. Thapa and S. Karki Table 1: Main result on DHCD test (n=13,800). Errors over 5 seeds; McNemar vs. reproduced baseline [9] (seed 0). ConfigurationParams Errors (5 seeds) Mean±std Acc % McNemar p Barnamala, distilled (TTA) 1.11M 34,39,35,39,36 36.6±2.1 99.735 0.345 Barnamala, distilled (clean) 1.11M 39,40,34,40,38 38.2±2.2 99.723 1.000 Barnamala, supervised1.11M 43,40,42,35,39 39.8±2.8 99.712 0.701 MallaNet [9] (repro)17.32M40n/a 99.710n/a Table 2: Published results on the standard 46-class DHCD test split (78,200 train / 13,800 test). All accuracies from primary sources. McNemar requires per-sample predictions; only available for the re-implemented baseline (Table 1). ModelYearParams Accuracy Architecture Barnamala (ours, mean 5 seeds) 20271.11 M 99.735 % Compact SE-ResNet Mishra et al. [11]2021 ≈39 M 99.72 % ResNet-85 MallaNet [9]202517.32 M 99.71 % Branch-merge CNN Kumar et al. [7]2023 not reported 99.68 % Vision Transformer Yadav et al. [15]2024 ≈0.4 M 99.21 % Modified LeNet-5 Acharya et al. [1]2015 ≈0.03 M 98.47 % CNN baseline 4.3 Comparison with Prior Work 0.07 % separates the top four, well within sampling noise on a 13,800-image test set. No other entry simultaneously clears 99.5 % accuracy and stays below the 2 M parameter mark. The next lightest model above that threshold is MallaNet at 17.32 M parameters [9], a 15.6× gap. 4.4 Efficiency This is 9.5× lower CPU latency and 14.8× fewer MACs, with no quantization or pruning and no loss in accuracy. 4.5 Distillation and Ensemble Ablation Effect of knowledge distillation and target cleanliness. Table 1 separates two questions: whether the student benefits from a teacher at all, and whether the way we generate teacher targets matters. In the primary configuration, distilled (TTA) targets, the student learns from soft labels generated by averaging each teacher’s predictions over a horizontal-flip pair of each training image (see Sec- tion 3.3); it ends at 36.6±2.1 errors. With distilled (clean) targets, the soft labels come from a single clean forward pass through a larger 15-teacher pool with no flip averaging, and the student reaches 38.2±2.2 errors. The supervised control, trained with no teacher signal at all, obtains 39.8±2.8 errors. Barnamala: Parameter-Efficient Devanagari Recognition7 Table 3: Efficiency vs. [9] (single 32×32 image; CPU latency single-thread, mean±std over 40 runs). ModelParams MACs Size Peak act. CPU lat. (b1) Tput Barnamala 1.11M 164M 4.4MB 2.8MB 8.28±0.20 ms 151 img/s MallaNet 17.32M 2434M 69.3MB 26.1MB 78.8±0.89 ms 13.6 img/s Ratio15.6× 14.8× 15.6× 9.5×9.5×11.1× Table 4: Teacher ensembles on DHCD test (n=13,800). Rows 1–2: scaling from 9 to 15 teachers does not move the 30-error plateau. Row 3: adding flip-TTA crosses p<0.05 but is methodologically invalid (Section 4.6). EnsembleErrors Acc % McNemar p p<0.05? 9 teachers, clean30 99.783 0.0755no 15 teachers, clean30 99.783 0.0755no 9 teachers, flip-TTA 28 99.797 0.029fragile Three errors separate the best distilled student and the supervised baseline – moderate, and none of the three McNemar p values approach 0.05 (Table 1). Distillation helps, but only a little. What’s more, is how unimportant the target recipe is. Students on the clean target and those on the TTA target are statisti- cally indistinguishable (38.2 vs. 36.6, with substantial seed overlap in the error distributions). A flipped-average target cannot beat a clean single forward pass through the same teacher. The useful signal is the teacher itself, not the extra target cleanup. For simplicity, we keep a small 9-teacher pool with a flipped average target as the primary configuration, but the results do not depend on that choice. Teacher ensemble size. Natural follow-up: Is the plateau of 30 errors an upper limit imposed by having only 9 teachers? To find out, we ran the ensemble as direct predictor on the test set, rather than as the source of training time soft targets, and compared 9 teachers to 15 teachers. The answer from Table 4 is straightforward. More teachers do not move the ceiling. Individual teachers disagree enough to make the test meaningful – the num- ber of clean errors per model ranges from 27 to 43. Yet both majority pools plateau at 30 errors (99.783%) with an identical McNemar p = 0.0755. Ensem- ble diversity is not the bottleneck: the remaining errors sit in an irreducible floor where additional models vote the same wrong answer. 4.6 Flip-TTA at Test Time If we average each teacher’s logits over the original and horizontally-flipped test image, the error drops to 28 (p = 0.029). This is the only result in our study that 8A. Thapa and S. Karki crosses the 0.05 threshold. This result is not treated as a positive result. Horizon- tal reflection of Devanagari produces characters that are not in the script. In the training distribution, such inversions are intentionally excluded (Section 3.4). Gain is also vulnerable. When applied to the individual teacher, the flip TTA causes errors in 88 and 102 at the most affected checkpoints, increasing from 30-43 in the clean baseline. It is only by chance at this split that the ensemble masks this difference. The student training-time targets (Section 3.3) also use flipped averaging, but use the training image as the target smoothing device. This is a clear application that is not affected by the invalidity of flip TTA as a test time predictor. Therefore, we exclude the results for p = 0.029 and report a clean ensemble of 30 errors as an upper criterion for teacher performance. 4.7 DHCD to CMATERdb Digit Transfer Do distilled students learn the structure of numbers, or just the visual habits of one collection pipeline? We probe this on CMATERdb 3.2.1 [2], an independently collected set of handwritten Devanagari numerals (digits 0-9, 10 classes). The two datasets have different scanner hardware, writer populations, paper stocks, and binarization pipelines, so the duplicate labels clearly arrive from different source distributions. With Hamming distance≤5 bits (minimum observed: 6 bits), no near-overlapping pairs were detected and contamination was ruled out. Before evaluation, each CMATERdb image is polarity-inverted to match the white-on-black convention of DHCD. Results. Table 5 (upper rows) summarises the transfer results. The DHCD- trained distilled student zero-shots at 76.6±3.7% on CMATERdb using the same 10-class head, versus 62.7% for a supervised baseline trained on DHCD digit images only—+14 percentage points under domain shift despite being tied on in-distribution DHCD. Freezing the backbone and fitting a fresh 10-class head (linear probe) reaches 85.4%; 30 epochs of fine-tuning at a low learning rate reaches 97.8%. There is one caveat: this experiment covers only the digit subset (10 of 46 DHCD classes). Nothing is claimed about transferring the full character inven- tory. Within that scope, after contamination and polarity control, the distilled student preserves more structure under dataset shift than the supervised DHCD- only baseline. 4.8 Corruption Robustness Three corruption families are evaluated: additive Gaussian noise, Gaussian blur, and contrast reductions. The five severity levels follow the ImageNet-C protocol adapted for 32×32 grayscale images. We report mean accuracy for each family and overall severity 1-5 (mCA). To maintain preprocessing symmetry, all models apply their own training-time normalisation to the same corrupted [0, 1] input image. Barnamala: Parameter-Efficient Devanagari Recognition9 Table 5: Transfer and corruption robustness. mCA = mean accuracy over sever- ities 1–5. MetricBarnamala MallaNet CMATERdb zero-shot (distilled) 76.6%n/a CMATERdb fine-tuned97.8%n/a Clean DHCD99.75% 99.71% mCA noise54.7%66.9% mCA blur97.0%37.6% mCA contrast75.5%11.5% mCA overall75.7% 38.7% Results. Barnamala gives about 2× better results overall (mCA 75.7% vs. 38.7%). The gap in blur and contrast is significant (97.0% vs. 37.6% for blur and 75.5% vs. 11.5% for contrast). This likely reflects that the baseline HFC design am- plifies blur and its fixed (0.5, 0.5) normalization is sensitive to global intensity changes caused by contrast corruption. There is one obvious reversal. For ad- ditive noise, the baseline is better (mCA 66.9% vs. 54.7%). Practitioners who target noisy deployment environments should note that the noise rankings are reversed. 4.9 Cross-Dataset Generalisation Both models are evaluated in zero-shot on two independently collected Devana- gari datasets, each with separate writer and scanner workflows from different collection years, without applying any fine-tuning. NHCD (Pant 2012). The Nepali Handwritten Character Dataset [12] covers the same 46-class inventory in 28×28 JPEG images; we polarity-invert and bilin- early resize to 32×32 before evaluation. Barnamala achieves 78.92% zero-shot (95.80% on digits, 72.33% on consonants). The baseline [9] collapses to 23.99% (58.85% on digits, 10.38% on consonants), a 55 p gap despite identical in- distribution performance on DHCD, with the +62 p consonant disparity being the most striking number. The dental/retroflex pair is the most obvious failure mode. Each of the four pairs (ta/t . a, tha/t . ha, da/d . a, dha/d . ha) collapses to near-zero accuracy, caused by interclass confusion between strokes with different rendering rules between Indian and Nepali handwriting traditions. This is a character-level domain gap that neither model can resolve. Outside this cluster, Barnamala and the baseline diverge sharply. Prashanth et al. 2021. After resizing to 32×32, Barnamala achieves 79.31% zero-shot [13] on this 22,500 image digit dataset. The baseline achieves 72.60%. These numbers track the NHCD digit results and the CMATERdb result from 10A. Thapa and S. Karki Table 6: Zero-shot cross-dataset accuracy. Both models trained on DHCD only; no fine-tuning on the target dataset. NHCD polarity-inverted before evaluation; Prashanth polarity matches DHCD. n NHCD = 10,260; n Prashanth = 22,500 (digit classes only). NHCD (Pant 2012, 46 classes) Prashanth 2021 ModelOverall Consonants Digits (10 digit classes) Barnamala (ours) 78.92% 72.33% 95.80% 79.31% MallaNet [9]23.99% 10.38% 58.85%72.60% Gap+54.9 p +61.9 p +36.9 p+6.7 p Section 4.7. The high rate of confusion of digit 5 with digit 7 is due to pure visual similarity in the Devanagari script and is not a training artifact. Matching accuracy within the distribution hides where the gaps are. Both out-of-distribution evaluations favor Barnamala. The corruption results in Sec- tion 4.8 show a similar pattern. This pattern across the three withheld assess- ments suggests that drawing from a diverse teacher ensemble transfers better across writer and scanner variations than the raw DHCD scores indicate. 5 Analysis: Why DHCD is Saturated 5.1 The Shared Error Floor Both the 15 student models and the reproduced baseline [9] fail on the same 11 images. No configuration classifies any of them correctly, regardless of architec- ture, training strategy, or random seed. If the same 11 image floors capture both a compact Student of 1.11 M pa- rameters and a baseline 15 times its size, the problem lies in the data. Outside this floor, the error is model-specific (average pairwise Jaccard similarity 0.48). Seeds shuffle which images fail, but the addition of ensembles or seeds reduces variance rather than bias imposed by the floor. 5.2 Confusable Character Pairs Script-level conflicts account for most of the remaining mistakes. Table 7 lists the top distillation-student confusion pairs aggregated across the 10 distillation seeds. We are not entirely sure what drives the ba/waw pair specifically (17 mutual confusions across all seeds); at 32×32 resolution both glyphs collapse to nearly identical stroke profiles and only a handful of pixels separate one from the other. waw / tabala (16) is second, tra / ba (15) third, with dha / gha (13) and da / dhaa (9) rounding out the top five. In 32×32 pixels, the characteristic strokes of these pairs span just a few pixels, sometimes less. DHCD provides neither high-resolution input nor stroke- level annotations, so the information needed to resolve them is not present in the image. Barnamala: Parameter-Efficient Devanagari Recognition11 12345678910111213141516 Models misclassifying the same test image (of 16) 0 5 10 15 20 25 Test images All-model floor: 11 images Top confused pairs ba -> waw: 17 waw -> tabala: 16 tra -> ba: 15 (a) Images missed by exactly k of the 16 models. Eleven are missed by all 16 simul- taneously: an irreducible floor. 05101520253035404550 Errors on DHCD test set (n=13,800); fewer is better → Students Ensembles References floor (11) p<0.01 (20) p<0.05 (25) MallaNet (40) Student distilled TTA Student distilled clean Student supervised 9-teacher ensemble 15-teacher ensemble excluded (semantically invalid) 9-teacher + flip-TTA MallaNet (baseline) Intrinsic floor Significance Frontier: No Configuration Crosses p<0.05 Mean ± 1 std (student configs)Excluded pointSignificance thresholds (b) Every configuration sits above the p<0.05 threshold (25 errors). The 11-error floor marks the hard lower bound. Fig. 2: Left: shared error floor across all 16 models. Right: significance frontier; 30-error clean ensemble and 36.6-error student mean both lie in the statistical parity region. Table 7: Top confusion pairs in the distilled student pool (aggregate across 10 distilled seeds on the DHCD test set). PairCross-confusionsScript note ba ↔ waw17Near-identical body, open vs. closed loop waw ↔ tabala16Shared curved baseline tra ↔ ba15Conjunct ligature vs. simple consonant dha ↔ gha13Looped ascender, subtle body curvature da ↔ dhaa9Single hairline distinguishes the pair 5.3 Significance Frontier Moving from statistical parity to a clear win over the reproduced baseline (40 errors) under an exact two-sided McNemar test [10] at α = 0.05 requires at most 25 errors (≥99.82% accuracy). Fewer than 15 additional correct classifications separate an inconclusive result from a significant one. A comfortable margin means ≤20 errors (≥99.855%), which would demand resolving almost every ambiguous glyph pair in Table 7. Every configuration hits the wall. The single best result is 28 errors, from the 9-teacher ensemble using (semantically invalid) flip-TTA at test time—excluded on methodological grounds (Section 4.6) because horizontal reflection produces characters that do not exist in Devanagari script and the gain does not survive when applied to individual teachers. The most defensible result is the clean ensemble at 30 errors (99.783%); the most defensible single seed reaches 34 errors. Both sit inside the statistical parity region. No lever tested in this study achieved a statistically significant im- provement over the baseline. Scaling from 9 to 15 teachers changes nothing. 12A. Thapa and S. Karki Five independent seeds with TTA-softened targets change nothing. This is not a failure of experimental design. It is what a saturated benchmark looks like. The frontier is a wall built from mislabeled images and inherently ambiguous glyphs, not a line that more computation can asymptotically approach. 5.4 Calibration Expected Calibration Error (ECE) [3] falls within the range of 0.13–0.16 across all Barnamala configurations. It is not noticeable for softmax classifiers without post-hoc temperature scaling. ECE does not covary with accuracy across seeds. A better seed will not calibrate better. The 11 floor images are confidently wrong, not merely uncertain, at every seed. The model assigns a high probability to the incorrect class, regardless of how the student is trained. 6 Discussion Scope of transfer results. The transfer experiment (Section 4.7) is deliberately narrow: it uses only the digit subset (10 of 46 DHCD classes). That choice reflects the available data, not a design preference. Extending the test to the full 36-consonant inventory would require a genuinely independent target set with DHCD’s label taxonomy; the apparent public candidates—principally the Acharya [1] and Pant datasets—are same-provenance splits of DHCD itself. Transferability claims here are therefore limited to digits, not consonants. Single-benchmark evaluation. The saturation claim is tied to DHCD’s particular conditions—its 32×32 resolution, concentrated label noise, and writer pool. A cleaner or more diverse consonant benchmark might expose capacity differences that DHCD no longer reveals. Efficiency versus accuracy. Compactness is the appropriate response to bench- mark saturation: when remaining errors are governed by data rather than model capacity, a larger network has little room to justify its cost. 7 Conclusion We have effectively reached the saturation point on DHCD. What remains can be attributed to label noise and handwriting ambiguity in the data itself, not model capacity. Throwing a bigger recognizer at it does not buy a better recognizer. At that point, compactness is no longer a compromise; it becomes the right operating point. Barnamala reaches 99.73 % accuracy with just 1.11 M parameters and achieves statistical equivalence with the leading large-scale baselines (McNemar p = 0.345) even without knowledge distillation. This means a parameter reduction of 15.6× and a CPU latency reduction of 9.5× without sacrificing accuracy. Why Barnamala: Parameter-Efficient Devanagari Recognition13 can’t we move forward? The same 11-error intrinsic floor breaks all tested con- figurations, teacher ensembles included. Ablation closes off every obvious escape route: distillation target cleanliness makes no difference. Ensemble size makes no difference. Test-time augmentation makes no difference. The floor is the ceiling. Two limitations are worth noting. The saturation claim is tied to the specific conditions of DHCD, namely 32×32 resolution, concentrated label noise, and its particular pool of writers. Cleaner or higher-resolution benchmarks may expose capacity differences that DHCD cannot. The transfer results only cover digit subset (10 out of 46 classes). Consonant-level transferability is out of scope here, as there are no independent consonant target sets that share the label taxonomy of DHCD. The practical implication for Devanagari recognition is to measure progress more honestly before claiming it. Without paired evaluation – McNemar or equiv- alent – a gain of a few hundredths of a percent is meaningless. The field needs fresher, harder benchmarks: DHCD can no longer discriminate between strong models. The next necessary step is high-resolution scanning or a completely independent pool of writers. References 1. Acharya, S., Pant, A.K., Gyawali, P.K.: Deep learning based large scale handwrit- ten Devanagari character recognition. In: 9th International Conference on Software, Knowledge, Information Management and Applications (SKIMA) (2015) 2. Das, N., Reddy, J.M., Sarkar, R., Basu, S., Kundu, M., Nasipuri, M., Basu, D.K.: A statistical-topological feature combination for recognition of handwritten numerals. Applied Soft Computing 12(8), 2486–2495 (2012) 3. Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. In: International Conference on Machine Learning (ICML) (2017) 4. He, K., Zhang, X., Ren, S., Sun, J.: Identity mappings in deep residual networks. In: European Conference on Computer Vision (ECCV) (2016) 5. Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. In: NIPS Deep Learning Workshop (2015) 6. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018) 7. Kumar, S., et al.: Vision transformer based Devanagari character recognition. In: AIP Conference Proceedings. vol. 2754, p. 160001 (2023). https://doi.org/10. 1063/5.0169520 8. Malla, S.R.: Devanagari digit recognition using quantum machine learning (2025) 9. Malla, S.R.: Mallanet residual branch merge convolutional neural network with ho- mogeneous filter capsules for Devanagari character recognition. Scientific Reports 16(1), 1175 (2025). https://doi.org/10.1038/s41598-025-30871-z 10. McNemar, Q.: Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12(2), 153–157 (1947) 11. Mishra, M., Choudhury, T., Sarkar, T.: Devanagari handwritten character recog- nition. In: 2021 IEEE India Council International Subsections Conference (INDIS- CON) (2021). https://doi.org/10.1109/INDISCON53343.2021.9582192 14A. Thapa and S. Karki 12. Pant, A.K., Panday, S.P., Joshi, S.R.: Off-line Nepali handwritten character recog- nition using multilayer perceptron and radial basis function neural networks. In: Third Asian Himalayas International Conference on Internet (AH-ICI). p. 1–5. IEEE (2012) 13. Prashanth, D.S., Mehta, R.V.K., Challa, N.P.: A multi-purpose dataset of Devana- gari script comprising of isolated numerals and vowels. Data in Brief 39, 107723 (2021). https://doi.org/10.1016/j.dib.2021.107723 14. Wilson, E.B.: Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association 22(158), 209–212 (1927) 15. Yadav, M., et al.: Recognizing off-line devanagari handwritten characters using modified LeNet-5. Procedia Computer Science 235, 799–809 (2024). https://doi. org/10.1016/j.procs.2024.04.076 16. Yun, S., Han, D., Oh, S.J., Chun, S., Choe, J., Yoo, Y.: CutMix: Regularization strategy to train strong classifiers with localizable features. In: IEEE/CVF Inter- national Conference on Computer Vision (ICCV) (2019) 17. Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond empirical risk minimization. In: International Conference on Learning Representations (ICLR) (2018)