Paper deep dive
WildHandBench: A Benchmark for Handwritten Text Understanding that Challenges MLLMs and Humans
Jun Zhang, Qiao Zhao, Cheng Cui, Jianying Qu, Zhongkai Sun, Jianwen Yang, Changda Zhou, ZhuoXin Liu, Shubin Han
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:While the top model on OmniDocBench now reaches 96.34% overall on printed-document parsing, the ability of current models to handle challenging handwritten documents remains largely uncharacterized. Existing benchmarks focus on isolated text or formulas, overlook handwritten tables and real-world degradation, and report aggregate accuracy without explaining why models fail. We present WildHandBench, a benchmark containing 500 handwritten documents across three structures (free text, tables, formulas), four languages, and nine real-world scenarios. We introduce a Prior-Driven Error (PDE) metric that quantifies whether errors originate from language priors rather than visual evidence. Evaluating 18 state-of-the-art models together with calibrated human baselines, we find: (1) the best model achieves only 71.85% overall; (2) humans outperform all models yet the gap is narrow (77.09% vs. 71.85%); and (3) model errors are qualitatively different from human errors -- 63-91% of model errors are prior-driven versus only 49% for humans, exposing systematic reliance on language priors that conventional accuracy metrics cannot capture.
Tags
Links
- Source: https://arxiv.org/abs/2608.22959v1
- Canonical: https://arxiv.org/abs/2608.22959v1
Trouble viewing inline? Open PDF directly â
Full Text
50,348 characters extracted from source content.
Expand or collapse full text
WildHandBench: A Benchmark for Handwritten Text Understanding that Challenges MLLMs and Humans Jun Zhang Qiao Zhao Cheng Cui Jianying Qu Zhongkai Sun Jianwen Yang Changda Zhou ZhuoXin Liu Shubin Han Abstract While the top model on OmniDocBench now reaches 96.34% overall on printed-document parsing, the ability of current models to handle challenging handwritten documents remains largely uncharacterized. Existing benchmarks focus on isolated text or formulas, overlook handwritten tables and real-world degradation, and report aggregate accuracy without explaining why models fail. We present WildHandBench, a benchmark containing 500 handwritten documents across three structures (free text, tables, formulas), four languages, and nine real-world scenarios. We introduce a Prior-Driven Error (PDE) metric that quantifies whether errors originate from language priors rather than visual evidence. Evaluating 18 state-of-the-art models together with calibrated human baselines, we find: (1) the best model achieves only 71.85% overall; (2) humans outperform all models yet the gap is narrow (77.09% vs. 71.85%); and (3) model errors are qualitatively different from human errorsâ63â91% of model errors are prior-driven versus only 49% for humans, exposing systematic reliance on language priors that conventional accuracy metrics cannot capture. Introduction Recent advances in multimodal large language models (MLLMs) have dramatically improved document understanding (7; 26). The top model on the OmniDocBench leaderboard now reaches 96.34% overall (26), approaching saturation on printed-document parsing. These remarkable results have gradually created a widespread perception that OCR is becoming a largely solved problem. We argue that this conclusion is premature. Nearly all evidence supporting this perception comes from printed documents with relatively regular layouts and clear visual appearances (23; 39; 38; 9). Handwritten documents tell a different story. Medical records, handwritten forms, classroom notes, historical archives, and personal correspondence remain substantially more difficult for todayâs MLLMs despite their impressive performance on printed documents. Unlike printed OCR, handwritten document understanding requires models to jointly resolve ambiguous handwriting, recover irregular document structures, distinguish visual evidence from language priors, and reason under genuine uncertainty (13; 4). Consequently, handwriting is not simply a more difficult OCR problem, but a distinct document understanding challenge. Surprisingly, current evaluation benchmarks provide limited insight into why handwritten document understanding remains unsolved. Existing handwritten benchmarks primarily evaluate isolated free-text recognition, while handwritten tables, naturally written formulas, and real-world document degradation remain largely unexplored. More fundamentally, existing evaluations focus almost exclusively on recognition accuracy (7; 26). They measure how often models fail, but provide little understanding of why they failâin particular, whether errors originate from poor visual perception or from excessive reliance on language priors. To address this gap, we introduce WildHandBench, a benchmark for handwritten document understanding. WildHandBench contains 500 handwritten documents spanning three document structures (free text, tables, and formulas), four language settings, and nine representative real-world scenarios. We also introduce a Prior-Driven Error (PDE) metric that characterizes whether model errors arise from language priors instead of visual evidence, enabling systematic analysis of hallucination-like recognition behaviors. Using WildHandBench, we conduct a comprehensive evaluation of 18 state-of-the-art MLLMs and OCR-oriented vision-language models under a unified evaluation protocol together with calibrated human baselines. Our study yields three key findings. First, handwritten document understanding remains far from solved: the strongest model achieves only 71.85% overall. Second, humans outperform all models (77.09% vs. 71.85%), yet the gap is narrowâtop models even surpass humans on text transcription under format-matching metrics. Third, model errors are qualitatively different from human errors: humans remain conservative on illegible content, whereas models confidently hallucinate fluent but unsupported text, with 63â91% of model errors classified as prior-driven compared to only 49% for humans. This exposes systematic reliance on language priors that conventional accuracy metrics cannot capture. Our contributions are summarized as follows: ⢠We propose WildHandBench, to our knowledge the first benchmark that jointly evaluates handwritten free text, tables, and formulas across multiple languages and diverse real-world scenarios, addressing key gaps in existing evaluation coverage. ⢠We introduce the Prior-Driven Error (PDE) metric, which quantifies whether model errors originate from language priors rather than visual evidence, enabling systematic analysis of hallucination-like recognition behaviors beyond conventional accuracy metrics. ⢠We conduct a comprehensive evaluation of 18 state-of-the-art MLLMs and OCR-oriented vision-language models together with calibrated human baselines, revealing the persistent handwriting gap, the narrow yet meaningful distance to human performance, and the systematic reliance of MLLMs on language priors. Related Work Handwritten document understanding. Handwritten document analysis has been studied for decades, yet existing benchmarks remain highly task-specific and fragmented. Most datasets focus on line-level handwritten text recognition, including IAM (22), RIMES (12), CASIA-HWDB (19), SCUT-EPT (41), and SCUT-HCCDoc (35), covering Latin and Chinese handwriting under relatively controlled settings. Scientific handwriting has been investigated through datasets such as NoTeS-Bank (27), while handwritten mathematical expression recognition is primarily benchmarked by CROHME (21) and MathWriting (8). In contrast, table understanding benchmarks, including PubTabNet (39), FinTabNet (38), and the ICDAR table competitions (9), are exclusively constructed from printed documents. Although these datasets have significantly advanced individual OCR tasks, they evaluate text, formulas, and tables independently and provide limited coverage of challenging handwritten documents with real-world degradation. Consequently, current handwritten benchmarks remain insufficient for evaluating handwritten documents as a unified document understanding problem. Benchmarking document understanding. The rapid development of multimodal large language models (MLLMs) has shifted document evaluation from isolated OCR tasks toward holistic document understanding. Benchmarks such as OmniDocBench (26) evaluate diverse document parsing capabilities under a unified framework, while Real5-OmniDocBench (40) further investigates robustness under real-world physical degradations through fine-grained factor-level evaluation. OCRBench (20) and OCRBench v2 (7) extend benchmark coverage to OCR-oriented multimodal capabilities, including text recognition, formula parsing, table understanding, and document visual question answering. These benchmarks demonstrate remarkable progress on printed documents, with the top model now reaching 96.34% overall on OmniDocBench (26). Nevertheless, these evaluations primarily report aggregate recognition accuracy, providing limited understanding of why handwritten documents remain substantially more challenging than printed documents. Prior-driven errors in vision-language models. Recent studies have shown that multimodal large language models (MLLMs) frequently generate outputs that are not fully grounded in visual evidence (13; 4). In OCR-related tasks, such hallucinations often manifest as fluent yet visually unsupported transcriptions, where language priors override ambiguous visual observations. To better characterize this phenomenon, Seong et al. (29) proposed PINK, which penalizes over-correction in handwritten mathematical expression recognition. Similarly, HunyuanOCR-1.5 (17) introduces CHAOS-Bench, which evaluates whether models faithfully preserve visually observed but semantically implausible text under synthetic character perturbations in printed documents. These studies consistently demonstrate that prior-driven recognition errors are systematic rather than incidental. However, existing evaluations remain restricted to either handwritten mathematical expressions or synthetically perturbed printed documents. To the best of our knowledge, no existing benchmark systematically measures prior-driven errors on naturally occurring handwritten documents spanning free text, tables, and formulas. Our proposed Prior-Driven Error (PDE) metric is designed to bridge this gap by quantifying the extent to which model errors are attributable to language priors instead of visual evidence in realistic handwritten document understanding. Table 1 summarizes representative benchmarks. Compared with previous datasets, WildHandBench is, to our knowledge, the first to unify handwritten free text, tables, and formulas within a single evaluation framework with calibrated human baselines and Prior-Driven Error analysis. Benchmark Lang. Handwritten Free Text Table Formula Real Degrad. Human Baseline IAM EN â â â â â â CASIA-HWDB ZH â â â â â â SCUT-EPT / HCCDoc ZH â â â â Partial â CROHME / MathWriting Symbol â â â â â â PubTabNet / FinTabNet EN â â â â â â ICDAR Table EN â â â â â â OCRBench (v1/ v2) Multi Partial â â â Partial â OmniDocBench ZH/EN Partial â â â Partial â Real5-OmniDocBench ZH/EN Partial â â â â â WildHandBench (Ours) ZH/EN â â â â â â Table 1: Comparison with representative benchmarks. WildHandBench is, to our knowledge, the first to jointly cover handwritten free text, tables, and formulas with real-scenario degradation and calibrated human baselines. WildHandBench: Benchmark Design WildHandBench jointly evaluates handwritten free text, tables, and formulas under realistic handwritten document scenarios. The benchmark contains 500 handwritten document images spanning three document structures, four language settings, and nine representative real-world scenarios. Figure 1 illustrates the overall construction pipeline. Benchmark Taxonomy To comprehensively characterize handwritten document understanding, every sample is organized along three dimensions. ⢠Language (4). WildHandBench covers four language settings, including Simplified Chinese (333, 66.6%), English (143, 28.6%), Traditional Chinese (13, 2.6%), and ChineseâEnglish mixed documents (11, 2.2%). ⢠Document Structure (3). The benchmark jointly evaluates three representative handwritten document structures: free text (367, 73.4%), tables (81, 16.2%), and formulas (52 images containing 278 individually annotated formula regions, 10.4%). ⢠Real-world Scenario (9). Samples are collected from nine representative scenarios, including literary writing, letters and notes, medical records, business and government documents, education, mathematical and scientific materials, classical calligraphy, daily miscellany, and historical archives. Benchmark Construction Figure 1: Overview of the WildHandBench construction pipeline. To maximize both dataset diversity and annotation reliability, handwritten documents are collected from two complementary sources. Offline handwritten manuscripts. Participants voluntarily contribute authentic handwritten materials covering multiple real-world domains. Since writers have direct access to the original content, reliable ground-truth transcriptions can be obtained from the source documents. Internet collection. Additional handwritten documents are collected from publicly available online resources where community discussions or decipherment references provide useful contextual information for annotation. After collection, all samples undergo automatic content-similarity scanning to remove duplicates, followed by manual inspection of ambiguous cases. Benchmark construction then proceeds in three stages. Stage 1: MLLM-assisted pre-annotation. Three state-of-the-art MLLMs independently transcribe every handwritten document. These predictions are not used as ground truth, but serve as reference suggestions for human annotators, helping identify ambiguous regions and improving annotation efficiency. Stage 2: Human annotation. A primary annotator produces the recognition ground truth for every sample, represented using Markdown for free text, LaTeX for formulas, and HTML <table> for handwritten tables. Additionally, each sample is annotated with difficulty ratings along six dimensions (visual degradation, cursiveness, linguistic ambiguity, structural complexity, domain knowledge, and multilingual mixing) to support future fine-grained analysis. During annotation, annotators consult all available evidence, including MLLM predictions, writer-provided transcripts, community discussions, and domain experts whenever necessary. Characters that remain impossible to identify after exhaustive verification are masked in the original image and excluded from evaluation. Stage 3: Independent verification. Every annotation is independently reviewed by a second annotator who performs the verification without access to the primary annotatorâs reasoning. Disagreements are resolved through discussion until consensus is reached. Only samples passing independent verification are included in the final benchmark. This multi-stage quality-control protocolâautomatic deduplication, MLLM-assisted annotation, independent double review, and consensus-based adjudicationâsubstantially reduces annotation inconsistency while ensuring reliable ground truth for subsequent evaluation. Evaluation Protocol Our evaluation protocol consists of three components. First, structure-specific recognition metrics measure transcription quality across free text, formulas, and tables. Second, the proposed Prior-Driven Error (PDE) metric quantifies whether recognition failures originate from language priors rather than visual evidence. Third, calibrated human baselines provide a reference point for characterizing how human and machine errors differ qualitatively. Recognition Evaluation Following OmniDocBench (26), different handwritten document structures are evaluated using structure-specific metrics. ⢠Free text. Character-level recognition quality is measured using normalized Edit Distance (Editâ ) (16). ⢠Formulas. Formula recognition is evaluated using CDM (Character Detection Metric, CDMâ ), which renders both prediction and ground truth before matching character bounding boxes (31). ⢠Tables. Table understanding is evaluated using TEDS (Tree Edit Distance Similarity, TEDSâ ), which jointly measures structural correctness and textual fidelity based on HTML representations (39). To facilitate comparison across different document structures, we report an overall score following OmniDocBench (26): Overall=(1âEdit)Ă100+CDM+TEDS3.Overall= (1-Edit)Ă 100+CDM+TEDS3. Prior-Driven Error Recognition accuracy alone cannot distinguish different types of recognition failures. Two models may achieve identical recognition accuracy while exhibiting very different behaviors: one may fail because it cannot correctly perceive handwritten content, whereas the other may generate visually unsupported yet linguistically plausible transcriptions due to excessive reliance on language priors. Since these two failure modes require different modeling improvements, WildHandBench explicitly quantifies prior-driven recognition errors through the proposed Prior-Driven Error (PDE) metric. For every prediction, model output is aligned with the corresponding ground truth using character-level alignment for free text and formulas, and cell-level alignment for tables. Only segments where prediction differs from the ground truth are considered. Each mismatched segment is then scored using an external reference language model (Qwen3-8B-Base (28)). If the model-generated segment has lower perplexity than the corresponding ground-truth segment, the error is regarded as prior-driven, indicating that the model replaces visually correct but less common content with a linguistically more probable alternative. Inserted content that has no corresponding ground truth is always regarded as prior-driven since it is generated solely from internal language priors. Formally, for a sample s, the Prior-Driven Error rate is computed as PDEâ(s)=âi[ppl(mi)<ppl(gi)]|mi|âi|mi|,PDE(s)= _i1[ppl(m_i)<ppl(g_i)]\,|m_i| _i|m_i|, where mim_i and gig_i denote the model prediction and the corresponding ground-truth segment, respectively. For inserted content, pplâ(gi)=âppl(g_i)=â. The summation is performed only over mismatched segments. Consequently, PDE measures the proportion of model errors attributable to language priors rather than visual perception, instead of reflecting overall recognition accuracy. The overall PDE score reported for each model is the arithmetic mean of the per-category PDE rates (text, table, formula). Human Baselines WildHandBench additionally establishes calibrated human baselines under the same evaluation protocol. Human participants receive exactly the same handwritten document images as the evaluated models and independently produce recognition outputs using the same target formats. Performance is evaluated using the identical metrics described above. To ensure a fair comparison, human participants perform single-pass recognition without access to any auxiliary resources. In particular, they are not allowed to consult MLLM predictions, writer-provided transcripts, community discussions, or domain experts. This setting differs intentionally from the benchmark construction process, where multiple evidence sources are integrated to establish reliable ground truth through consensus-based annotation. Under this protocol, human participants achieve an overall score of 77.09%, which is higher than the best MLLM (Gemini 3.1 Pro, 71.85%). Detailed per-category comparisons are presented in the Experiments section. Experiments Evaluated Models Figure 2: Qualitative comparison of handwritten table parsing across general-purpose models, OCR-specialized models, and the human baseline. The right panel illustrates three error types: table structure errors (E1), table content errors (E2), and prior-driven errors (E3). Performance Evaluation (%) Prior-Driven Error (%) Type Model Params Text Editâ Table TEDSâ Formula CDMâ Overall â Textâ Tableâ Formulaâ Overallâ Proprietary Claude Opus 4.8 â 53.30 52.27 74.11 57.70 72.78 59.32 91.66 74.59 Gemini 3.5 Flash â 27.10 53.63 78.22 68.25 71.36 71.91 94.36 79.21 Gemini 3.1 Pro â 24.32 60.44 79.42 71.85 71.19 66.56 94.77 77.51 Open General InternVL3.5 241B-A28B 46.80 41.22 64.00 52.80 72.62 66.38 90.03 76.34 Qwen3-VL 235B-A22B 34.30 51.81 69.06 62.19 70.01 62.18 91.33 74.51 Qwen3.5-Plus 397B-A17B 34.80 57.68 73.77 65.55 71.74 70.00 91.71 77.82 Kimi-K3 2.8T-104B 26.07 58.99 70.50 67.81 65.29 60.32 90.21 71.94 Open OCR DeepSeek-OCR2 3B-A0.5B 91.70 1.77 18.33 9.46 84.06 90.00 98.35 90.80 MinerU-2.5 1.2B 83.80 41.19 36.56 31.33 73.80 55.21 92.16 73.72 Unlimited-OCR 3B-A0.5B 64.90 34.41 51.62 40.38 61.67 55.16 89.47 68.77 Dots.ocr 3B 56.68 34.70 53.17 43.73 70.29 59.75 90.98 73.67 Dots.mocr 3B 49.54 36.60 62.85 49.97 63.23 59.56 89.73 70.84 HunyuanOCR 1.0B 38.50 47.02 45.81 51.44 67.13 55.55 92.08 71.59 MinerU2.5-Pro 1.2B 59.80 53.16 61.79 51.73 61.23 47.46 92.80 67.16 Qianfan-OCR 4B 50.40 45.22 61.40 52.07 64.07 61.29 94.04 73.13 FireRed-OCR 2B 49.00 44.30 63.11 52.82 64.70 64.25 93.10 74.02 GLM-OCR 0.9B 54.20 57.38 59.01 54.06 57.02 41.92 90.89 63.28 PaddleOCR-VL-1.6 0.9B 41.52 52.41 51.75 54.21 52.59 53.01 87.83 64.48 Human Human â 30.08 74.78 86.56 77.09 66.96 34.48 46.36 49.27 Table 2: Recognition performance and prior-driven error for all evaluated models and the human baseline. We evaluate 18 models grouped into three categories: proprietary VLMs (Claude Opus 4.8 (2), Gemini 3.1 Pro (10), Gemini 3.5 Flash (11)), open-source general VLMs (InternVL3.5 (25), Qwen3-VL (3), Qwen3.5-Plus (1), Kimi-K3 (15)), and open-source OCR-focused VLMs (DeepSeek-OCR-V2 (32), MinerU2.5 (24), MinerU2.5-Pro (30), Unlimited-OCR (34), Dots.ocr (18), Dots.mocr (37), HunyuanOCR (14), Qianfan-OCR (5), FireRed-OCR (33), GLM-OCR (6), PaddleOCR-VL-1.6 (36)). All models are evaluated with an identical post-processing pipeline. General VLMs use the standard OmniDocBench prompt template (26), while OCR-focused models use their respective recommended prompt settings or APIs. Models are selected to represent the current state of the art in document understanding, with reference to the OmniDocBench leaderboard (26). Main Results Table 2 presents the main results. The most striking finding is the performance degradation from printed to handwritten documents: models that exceed 90% on OmniDocBench (26) drop to 71.85% at best on WildHandBench, confirming a persistent âhandwritten gap.â Furthermore, OCR-focused modelsâwhich routinely outperform general VLMs on printed-document benchmarksâfall behind general VLMs on wild handwriting. The top six models on WildHandBench are all general-purpose, suggesting that larger model capacity and more diverse training data enable stronger generalization to the high variability of handwritten content, while OCR-focused models are specialized for the regularity of printed layouts. Tables remain the most challenging category: even the best TEDS score (60.44) indicates nearly 40% structural and content mismatch, and table scores show the smallest cross-model variance among functional models. Text recognition shows the widest spread, reflecting large differences in character-level fidelity. Gemini 3.1 Pro leads in all three categories and overall, though Kimi-K3 is competitive on text edit distance (26.07 vs. 24.32) and Gemini 3.5 Flash on formula CDM (78.22 vs. 79.42). Human vs. Model Analysis The human baseline achieves an overall score of 77.09%âhigher than the best MLLM (Gemini 3.1 Pro, 71.85%) by 5.24 absolute points. While this confirms that human performance remains the upper bound, the relatively modest gap indicates that wild handwriting challenges humans and models alike. The per-category breakdown reveals where models still lag behind humans and where they have caught up. On text, the best model actually surpasses humans (Edit: 30.08 vs. 24.32), indicating that MLLMs produce more format-compliant transcriptions even though humans may achieve superior character-level recognition. On tables (TEDS: 74.78 vs. 60.44), humans substantially outperform all models by over 14 absolute points, reflecting superior structural understanding of irregular handwritten tables. On formulas (CDM: 86.56 vs. 79.42), humans also lead, though the gap is smaller (7.14 points), indicating that formula recognition in current MLLMs is relatively mature. Crucially, human errors are qualitatively different from model errors. The PDE analysis shows that human PDE is 49.27%, substantially lower than all evaluated models (minimum 63.28%), indicating that human errors are more evenly split between visual misrecognition and prior-driven substitution. Models, in contrast, produce disproportionately prior-driven errorsâgenerating fluent but visually unsupported output where humans tend toward conservative partial transcriptions. This qualitative gap in failure modes persists despite the narrowing quantitative gap in overall accuracy. Error Pattern Analysis Table 2 also reports the prior-driven error rate decomposed by content structure; Figure 2 provides qualitative examples of table parsing errors across model types and the human baseline. Among models with functional visual encoders, PDE rates range from 63% to 79% without a simple correlation to accuracy. Notably, general VLMs with strong language capabilities show relatively high rates (Gemini 3.5 Flash: 79.21%, Qwen3.5-Plus: 77.82%, Gemini 3.1 Pro: 77.51%), while the lowest PDE rates belong to certain compact OCR-focused models (GLM-OCR: 63.28%, PaddleOCR-VL-1.6: 64.48%). This reveals a nuanced relationship between model capability and error type. At the extreme, models with severely limited visual decoding produce almost exclusively prior-driven errors. Among better-performing models, powerful language priors act as a double-edged sword: they boost accuracy by correctly inferring ambiguous characters, but when errors do occur, those errors are disproportionately prior-drivenâthe modelâs strong language capability causes it to confidently generate plausible alternatives rather than fail silently. Conversely, the models with the lowest PDE rates (GLM-OCR, PaddleOCR-VL) are compact models that produce a larger proportion of visual misrecognition errors (e.g., confusing similar-looking characters) rather than generating fluent but unsupported text. The per-structure breakdown reveals that formula errors are almost entirely prior-driven across all models (87â98%), likely because LaTeX syntax is inherently low-perplexityâeven legitimate format differences satisfy the ppl criterion. Text errors show the widest spread (52â84%), reflecting the greatest variation in model reliance on language priors. Table errors are moderate (42â90%), with weaker models showing higher rates. We note limitations of this metric: it characterizes error type, not error quantityâa high prior-driven rate does not mean a model is less reliable, only that its errors are more systematic. The classification depends on a specific reference language model (Qwen3-8B-Base); a different model may shift boundary cases. Discussion The findings of WildHandBench demonstrate that handwritten document understanding remains unsolved: models exceeding 90% on OmniDocBench (26) drop to 71.85% at best on WildHandBench, and this degradation is non-uniform across document structures. More critically, the PDE analysis reveals that model and human errors are qualitatively differentâ63â79% of model errors among functional models originate from language priors, compared to 49% for humans. This means that model errors are systematically more dangerous: fluent, confident, and unsupported by visual evidence. These findings have implications for safety-critical applications such as medical records, financial documents, and legal archives, where prior-driven errors are particularly difficult to detect (4). Future models should optimize not only recognition accuracy but also visual grounding and calibrated uncertainty, learning to express uncertainty rather than confidently generating plausible transcriptions when visual evidence is insufficient. Limitations Benchmark scale and coverage. Although WildHandBench contains carefully curated handwritten documents with high-quality annotations, it currently consists of 500 samples and should not be interpreted as a population-level estimate of handwritten OCR performance. The relatively small formula (52 images) and table (81 images) subsets limit statistical power for distinguishing highly competitive models. Furthermore, the benchmark focuses primarily on Chinese and English handwriting. Future versions will expand both language coverage and dataset scale. Annotation and human evaluation. Part of the benchmark is collected from publicly available Internet resources, which may introduce source bias despite careful manual screening and deduplication. Certain handwritten scenarios, such as historical calligraphy or severely degraded manuscripts, inherently involve subjective interpretation, making a universally correct transcription difficult to define even under consensus-based annotation. The reported human baseline should therefore be regarded as a calibrated reference rather than an absolute upper bound. Moreover, all participating annotators are native Chinese speakers, which may underestimate achievable human performance on highly cursive or stylistically diverse English handwriting. PDE metric. The PDE metric depends on an external reference language model (Qwen3-8B-Base (28)) for perplexity estimation, and alternative reference models may lead to different classifications for ambiguous boundary cases. Moreover, PDE is designed to characterize error type rather than quantityâa higher PDE does not imply lower model reliability, only that a larger proportion of errors originate from language priors rather than visual perception. Conclusion We presented WildHandBench, a benchmark for handwritten document understanding that jointly evaluates free text, tables, and formulas with a unified protocol combining recognition metrics, human baselines, and Prior-Driven Error analysis. Experiments on 18 models reveal a persistent handwritten gap (best model 71.85% vs. human 77.09%), with model errors qualitatively different from human errorsâ63â79% prior-driven for models vs. 49% for humans. References Alibaba Cloud (2026) Alibaba Cloud Qwen3.5-Plus. Note: Model Studio release documentationAccessed 2026-07-27 External Links: Link Cited by: Evaluated Models. Anthropic (2026) Anthropic Introducing Claude Opus 4.8. Note: Official model releaseAccessed 2026-07-27 External Links: Link Cited by: Evaluated Models. Bai et al. (2025) S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, et al. Qwen3-VL technical report. arXiv preprint arXiv:2511.21631. External Links: Document Cited by: Evaluated Models. Bai et al. (2024) Z. Bai, P. Wang, T. Xiao, T. He, Z. Han, Z. Zhang, and M. Z. Shou Hallucination of multimodal large language models: a survey. arXiv preprint arXiv:2404.18930. External Links: Document Cited by: Introduction, Related Work, Discussion. Dong et al. (2026) D. Dong, M. Zheng, D. Xu, C. Luo, B. Zhuang, Y. Li, R. He, H. Wang, W. Zhang, W. Wang, et al. Qianfan-OCR: a unified end-to-end model for document intelligence. arXiv preprint arXiv:2603.13398. External Links: Document Cited by: Evaluated Models. Duan et al. (2026) S. Duan, Y. Xue, W. Wang, Z. Su, H. Liu, S. Yang, G. Gan, G. Wang, Z. Wang, S. Yan, et al. GLM-OCR technical report. arXiv preprint arXiv:2603.10910. External Links: Document Cited by: Evaluated Models. Fu et al. (2025) L. Fu, B. Yang, Z. Kuang, J. Song, Y. Li, L. Zhu, Q. Luo, X. Wang, H. Lu, M. Huang, Z. Li, G. Tang, B. Shan, C. Lin, Q. Liu, B. Wu, H. Feng, H. Liu, C. Huang, J. Tang, W. Chen, L. Jin, Y. Liu, and X. Bai OCRBench v2: an improved benchmark for evaluating large multimodal models on visual text localization and reasoning. arXiv preprint arXiv:2501.00321. External Links: Document Cited by: Introduction, Introduction, Related Work. Gervais et al. (2024) P. Gervais, A. Fadeeva, and A. Maksai MathWriting: a dataset for handwritten mathematical expression recognition. arXiv preprint arXiv:2404.10690. External Links: Document Cited by: Related Work. GĂśbel et al. (2013) M. GĂśbel, T. Hassan, E. Oro, and G. Orsi ICDAR 2013 table competition. In 2013 12th International Conference on Document Analysis and Recognition, p. 1449â1453. External Links: Document Cited by: Introduction, Related Work. Google DeepMind (2026a) Google DeepMind Gemini 3.1 Pro: a smarter model for your most complex tasks. Note: Official model releaseAccessed 2026-07-27 External Links: Link Cited by: Evaluated Models. Google DeepMind (2026b) Google DeepMind Gemini 3.5 Flash model card. Note: Official model cardAccessed 2026-07-27 External Links: Link Cited by: Evaluated Models. Grosicki et al. (2009) E. Grosicki, M. CarrĂŠ, J. Brodin, and E. Geoffrois Results of the RIMES evaluation campaign for handwritten mail processing. In 2009 10th International Conference on Document Analysis and Recognition, p. 941â945. External Links: Document Cited by: Related Work. Guan et al. (2024) T. Guan, F. Liu, X. Wu, R. Xian, Z. Li, X. Liu, X. Wang, L. Chen, F. Huang, Y. Yacoob, D. Manocha, and T. Zhou HallusionBench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 14375â14385. External Links: Link Cited by: Introduction, Related Work. Hunyuan Vision Team et al. (2025) Hunyuan Vision Team, P. Lyu, X. Wan, G. Li, S. Peng, W. Wang, L. Wu, H. Shen, Y. Zhou, C. Tang, et al. HunyuanOCR technical report. arXiv preprint arXiv:2511.19575. External Links: Document Cited by: Evaluated Models. Kimi Team (2026) Kimi Team Kimi K3: open frontier intelligence. arXiv preprint arXiv:2607.24653. External Links: Document Cited by: Evaluated Models. Levenshtein (1966) V. I. Levenshtein Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady 10 (8), p. 707â710. Cited by: 1st item. Li et al. (2026) G. Li, X. Wan, S. Peng, W. Wang, H. Feng, Y. Du, B. Wu, Z. Ruan, Z. Lu, L. Wu, P. Lyu, H. Shen, Z. Lin, S. Hu, J. Yang, H. Wen, G. Yu, H. Liu, B. Wang, C. Ma, H. Hu, C. Zhang, and Y. Zhou HunyuanOCR-1.5: making lightweight OCR VLMs faster and better. arXiv preprint arXiv:2607.04884. External Links: Document Cited by: Related Work. Li et al. (2025) Y. Li, G. Yang, H. Liu, B. Wang, and C. Zhang dots.ocr: multilingual document layout parsing in a single vision-language model. arXiv preprint arXiv:2512.02498. External Links: Document Cited by: Evaluated Models. Liu et al. (2011) C. Liu, F. Yin, D. Wang, and Q. Wang CASIA online and offline chinese handwriting databases. In 2011 International Conference on Document Analysis and Recognition, p. 37â41. External Links: Document Cited by: Related Work. Liu et al. (2024) Y. Liu, Z. Li, M. Huang, B. Yang, W. Yu, C. Li, X. Yin, C. Liu, L. Jin, and X. Bai OCRBench: on the hidden mystery of OCR in large multimodal models. Science China Information Sciences 67, p. 220102. External Links: Document Cited by: Related Work. Mahdavi et al. (2019) M. Mahdavi, R. Zanibbi, H. Mouchère, C. Viard-Gaudin, and U. Garain ICDAR 2019 CROHME + TFD: competition on recognition of handwritten mathematical expressions and typeset formula detection. In 2019 International Conference on Document Analysis and Recognition, p. 1533â1538. External Links: Document Cited by: Related Work. Marti and Bunke (2002) U. Marti and H. Bunke The IAM-database: an english sentence database for offline handwriting recognition. International Journal on Document Analysis and Recognition 5 (1), p. 39â46. External Links: Document Cited by: Related Work. Mathew et al. (2021) M. Mathew, D. Karatzas, and C. V. Jawahar DocVQA: a dataset for VQA on document images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, p. 2200â2209. External Links: Document Cited by: Introduction. Niu et al. (2025) J. Niu, Z. Liu, Z. Gu, B. Wang, L. Ouyang, Z. Zhao, T. Chu, T. He, F. Wu, Q. Zhang, et al. MinerU2.5: a decoupled vision-language model for efficient high-resolution document parsing. arXiv preprint arXiv:2509.22186. External Links: Document Cited by: Evaluated Models. OpenGVLab (2025) OpenGVLab InternVL3.5. Note: Official release blogAccessed 2026-07-27 External Links: Link Cited by: Evaluated Models. Ouyang et al. (2025) L. Ouyang, Y. Qu, H. Zhou, J. Zhu, R. Zhang, Q. Lin, B. Wang, Z. Zhao, M. Jiang, X. Zhao, J. Shi, F. Wu, P. Chu, M. Liu, Z. Li, C. Xu, B. Zhang, B. Shi, Z. Tu, and C. He OmniDocBench: benchmarking diverse PDF document parsing with comprehensive annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 24838â24848. External Links: Link Cited by: Introduction, Introduction, Related Work, Recognition Evaluation, Recognition Evaluation, Evaluated Models, Evaluated Models, Main Results, Discussion. Pal et al. (2025) A. Pal, S. Biswas, A. Das, A. Lodh, P. Banerjee, S. Chattopadhyay, D. Karatzas, J. Llados, and C. V. Jawahar NoTeS-Bank: benchmarking neural transcription and search for scientific notes understanding. arXiv preprint arXiv:2504.09249. External Links: Document Cited by: Related Work. Qwen Team (2025) Qwen Team Qwen3 technical report. arXiv preprint arXiv:2505.09388. External Links: Document Cited by: Prior-Driven Error, Limitations. Seong et al. (2026) J. Seong, W. Liermann, M. Kim, J. Shin, and S. Lim When VLMs âfixâ students: identifying and penalizing over-correction in the evaluation of multi-line handwritten math OCR. arXiv preprint arXiv:2604.22774. External Links: Document Cited by: Related Work. Wang et al. (2026) B. Wang, T. He, L. Ouyang, F. Wu, Z. Zhao, T. Chu, Y. Qu, Z. Jin, W. Zeng, Z. Miao, et al. MinerU2.5-Pro: pushing the limits of data-centric document parsing at scale. arXiv preprint arXiv:2604.04771. External Links: Document Cited by: Evaluated Models. Wang et al. (2025) B. Wang, F. Wu, L. Ouyang, Z. Gu, R. Zhang, R. Xia, B. Shi, B. Zhang, and C. He Image over text: transforming formula recognition evaluation with character detection matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 19681â19690. External Links: Document Cited by: 2nd item. Wei et al. (2026) H. Wei, Y. Sun, and Y. Li DeepSeek-OCR 2: visual causal flow. arXiv preprint arXiv:2601.20552. External Links: Document Cited by: Evaluated Models. Wu et al. (2026) H. Wu, H. Lou, X. Li, Z. Zhong, Z. Sun, P. Chen, X. Zhou, K. Zuo, Y. Chen, X. Tang, et al. FireRed-OCR technical report. arXiv preprint arXiv:2603.01840. External Links: Document Cited by: Evaluated Models. Yin et al. (2026) Y. Yin, H. Liu, Q. Xie, C. Liu, S. Yang, S. Wang, Z. Liu, H. Zou, J. Chen, S. Wei, et al. Unlimited OCR works. arXiv preprint arXiv:2606.23050. External Links: Document Cited by: Evaluated Models. Zhang et al. (2020) H. Zhang, L. Liang, and L. Jin SCUT-HCCDoc: a new benchmark dataset of handwritten chinese text in unconstrained camera-captured documents. Pattern Recognition 108, p. 107559. External Links: Document Cited by: Related Work. Zhang et al. (2026) Z. Zhang, H. Liu, S. Liang, Y. Zhang, Y. Xiang, J. Liu, T. Sun, M. Lin, Y. Zhang, C. Zhou, T. Gao, C. Cui, Y. Liu, D. Yu, and Y. Ma PaddleOCR-VL-1.6: expanding the frontier of document parsing with under-optimized region refinement and progressive post-training. arXiv preprint arXiv:2606.03264. External Links: Document Cited by: Evaluated Models. Zheng et al. (2026) H. Zheng, Y. Li, K. Zhang, L. Xin, G. Zhao, H. Liu, J. Chen, J. Lou, J. Qiu, Q. Fu, et al. Multimodal OCR: parse anything from documents. arXiv preprint arXiv:2603.13032. External Links: Document Cited by: Evaluated Models. Zheng et al. (2021) X. Zheng, D. Burdick, L. Popa, X. Zhong, and N. X. R. Wang Global table extractor (GTE): a framework for joint table identification and cell structure recognition using visual context. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, p. 697â706. External Links: Link Cited by: Introduction, Related Work. Zhong et al. (2020) X. Zhong, E. ShafieiBavani, and A. Jimeno Yepes Image-based table recognition: data, model, and evaluation. In Computer Vision â ECCV 2020, p. 564â580. External Links: Document Cited by: Introduction, Related Work, 3rd item. Zhou et al. (2026) C. Zhou, Z. Gao, X. Wang, T. Gao, C. Cui, J. Tang, and Y. Liu Real5-OmniDocBench: a full-scale physical reconstruction benchmark for robust document parsing in the wild. arXiv preprint arXiv:2603.04205. External Links: Document Cited by: Related Work. Zhu et al. (2019) Y. Zhu, Z. Xie, L. Jin, X. Chen, Y. Huang, and M. Zhang SCUT-EPT: new dataset and benchmark for offline chinese text recognition in examination paper. IEEE Access 7, p. 370â382. External Links: Document Cited by: Related Work. Supplementary Material Section A provides additional dataset construction details beyond those in the main paper. Section B presents qualitative PDE (Prior-Driven Error) examples from WildHandBench. Section C gives complete evaluation details including the full model list, prompt templates, and inference configuration. A. Dataset Details This section expands on the dataset construction summarized in the main paper. We report only details not already covered there, including per-source sample counts, the joint categoryâlanguage distribution, per-scenario counts, and the annotation and image-processing specifications. A.1 Data Collection Sources The 500 samples are drawn from two complementary channels: ⢠Offline handwritten manuscripts (141 samples, 28.2%). Voluntary contributions from project members and acquaintances, concentrated in the education and medical domains (classroom notes, homework, medical records, prescriptions), plus hand-transcribed article abstracts, reading notes, and letters to broaden writing styles. ⢠Internet collection (359 samples, 71.8%). Publicly available handwritten images covering long-tail genres: classical poetry and calligraphy, historical archival registers, old newspapers, and official forms. A.2 Category Ă Language Distribution The main paper reports the marginal counts for document structure and language separately. Table 3 gives the full joint distribution. Table 3: Category Ă Language joint distribution. Category zh_hans en zh_hant zh_en_mix Total Text 251 99 12 7 369 Table 62 16 0 1 79 Formula 20 29 0 3 52 Total 333 144 12 11 500 A.3 Scenario Distribution The main paper enumerates the nine scenarios; Table 4 adds their per-scenario sample counts. Table 4: Per-scenario sample counts. Scenario Count Literary essays 93 Letters & notes 81 Medical & health 62 Education & learning 61 Business documents 58 Math & science formulas 55 Classical poetry & calligraphy 41 Daily life notes 32 Historical archives 17 Total 500 A.4 Difficulty Annotation Protocol The six difficulty dimensions defined in the main paper are each scored independently by 5 reviewers per sample, and per-dimension scores are averaged across reviewers. These ratings support fine-grained analysis but are not used in the aggregate benchmark scores. A.5 Annotation Format Details Beyond the target formats stated in the main paper (Markdown for free text, LaTeX for formulas, HTML <table> for tables), formula regions are additionally annotated with polygon coordinates (closed polygons rather than axis-aligned boxes) to accommodate slanted handwritten layouts, with each polygon paired one-to-one with its LaTeX transcription. Table annotations carry full structural markup (rowspan, colspan, header hierarchy). A.6 Image Processing All images are stored in PNG format with the following specifications: ⢠Maximum file size: 1 MB (palette optimization applied where needed). ⢠EXIF normalization: rotation tags are applied and stripped. ⢠Naming convention: hardhand_<UUID_no_dashes>.png (32-character hex UUID). B. Qualitative PDE Examples Figure 3: Classical poem (calligraphy). A handwritten copy of Su Shiâs Chibi Nostalgia contains non-standard character orderings. Gemini âcorrectsâ the handwriting to the memorized textbook versionâswapping the non-standard two-character ordering (tĂĄo lĂ ng) back to the canonical order (lĂ ng tĂĄo). This is a classic canonical-text prior: the model overwrites the actual handwritten content with a remembered standard version. The human baseline faithfully preserves the non-standard forms and marks only truly illegible characters with â?â. Figure 4: Personal letter. Gemini substitutes âprefixâ with âparameterââa semantic-field substitution in which the model replaces the written word with a semantically related term drawn from its priors rather than transcribing the source faithfully. Figure 5: Clinical diagnosis. The GT diagnosis is âacute mesenteric lymphadenitis / acute enteritis / terminal ileitis.â Gemini outputs a completely different yet medically plausible diagnosis (âcolonic diverticulitis / colonic polyps / terminal ileitisâ; only the third item matches). This is the most extreme form of PDE: medical domain-knowledge priors fabricate a coherent but factually wrong diagnosis. The human baseline honestly writes â?â for illegible portions. Figure 6: Traditional Chinese medicine prescription. Gemini applies a prior-driven substitution, replacing a herb name with a more frequently prescribed alternative. Figure 7: Political study notes. Gemini rewrites into more formal policy terminology (âself-relianceâ â âself-strengtheningâ) and swaps the character order inside a four-character phrase (âachievement makes talentâ â âtalent makes achievementâ)âboth linguistically valid but not matching the source. This cleanly contrasts high PDE (semantic rewriting) against low PDE (visual misreading). Figure 8: Childcare training reflection. Gemini fabricates an entirely different topic: âdaily cleaning skills trainingâ becomes ârecent first-aid training,â and âinfant bathing / umbilical careâ becomes âinfant airway obstruction careââthe whole passage topic is replaced. PaddleOCR-VL exhibits character-level noise but preserves the correct topic keywords (âdaily cleaning,â âbathingâ). The human baseline matches the GT exactly. Figure 9: Elementary-school essay. The student wrote âdoing heart-warming things in places you cannot see.â Gemini polishes it to âdoing heart-warming things when you most need helpââsemantically âbetterâ but not the original text (PDE). Both Gemini and PaddleOCR-VL are effectively âpolishingâ the childâs composition. Figure 10: Mathematical derivation (Lagrange multipliers). Gemini corrects the awkward phrase âin the same direction of proportionalâ to âin the same direction & proportional.â PaddleOCR-VL misreads âproportionalâ as âprepatianâ and garbles the âgâ g matrix line, but shows no semantic rewriting. C. Complete Evaluation Details C.1 Model List & Access Details Table 5 lists all evaluated models with their access method and model identifier. Table 5: Complete list of evaluated models. Model Name Type Access Model ID / Version GPT-5.4 General VLM API (OpenAI-compat) gpt-5.4 Gemini-3.1-Pro General VLM API (OpenAI-compat) gemini-3.1-pro-preview Gemini-3.5-Flash General VLM API (OpenAI-compat) gemini-3.5-flash Claude-Opus-4-8 General VLM API (OpenAI-compat) claude-opus-4-8 Qwen3.5-Plus General VLM API (OpenAI-compat) qwen3.5-plus Qwen3-VL General VLM API (vLLM) qwen3-vl-235b-a22b-thinking InternVL3.5 General VLM API internvl3.5-241b-a28b Kimi-K3 General VLM API (Anthropic fmt) kimi-k3 Qianfan-OCR OCR-focused API (Baidu) qianfan-ocr Hunyuan-OCR OCR-focused Local (vLLM) tencent/HunyuanOCR GLM-OCR OCR-focused API (ZhipuAI) glm-ocr PaddleOCR-VL OCR-focused API (async job) PaddleOCR-VL-1.6 DeepSeek-OCR-v2 OCR-focused Local (vLLM) deepseek-ai/DeepSeek-OCR FireRed-OCR OCR-focused Local (vLLM) FireRed-OCR Unlimited-OCR OCR-focused Local (vLLM) Unlimited-OCR MinerU-2.5 Document parser Local (vLLM) opendatalab/MinerU2.5-2509-1.2B MinerU-2.5-Pro Document parser Local (vLLM) opendatalab/MinerU2.5-Pro C.2 Prompt Templates All general-purpose VLMs (GPT-5.4, Gemini-3.1-Pro, Gemini-3.5-Flash, Claude-Opus-4-8, Qwen3.5-Plus, Qwen3-VL, InternVL3.5, Kimi-K3) use the standard OmniDocBench prompt. For all OCR-focused models (Qianfan-OCR, Hunyuan-OCR, GLM-OCR, PaddleOCR-VL, DeepSeek-OCR-v2, FireRed-OCR, Unlimited-OCR) and document parsers (MinerU-2.5, MinerU-2.5-Pro), we use the default API parameters when calling their APIs, or the default settings of the model when deployed locallyâno user-supplied prompt is overridden. OmniDocBench prompt (default). ⏠You are an AI assistant specialized in converting PDF images to Markdown format. Please follow these instructions for the conversion: 1. Text Processing: - Accurately recognize all text content in the PDF image without guessing or inferring. - Convert the recognized text into Markdown format. - Maintain the original document structure, including headings, paragraphs, lists, etc. 2. Mathematical Formula Processing: - Convert all mathematical formulas to LaTeX format. - Enclose inline formulas with \( \). For example: This is an inline formula \( E = mc^2 \) - Enclose block formulas with \[ \]. For example: \[ -b Âą b^2 - 4ac2a \] 3. Table Processing: - Convert tables to HTML format. - Wrap the entire table with <table> and </table>. 4. Figure Handling: - Ignore figures content in the PDF image. Do not attempt to describe or convert images. 5. Output Format: - Ensure the output Markdown document has a clear structure with appropriate line breaks between elements. - For complex layouts, try to maintain the original documentâs structure and format as closely as possible. Please strictly follow these guidelines to ensure accuracy and consistency in the conversion. Your task is to accurately convert the content of the PDF image into Markdown format without adding any extra explanations or comments. C.3 Inference Configuration All models share a common retry and timeout strategy: ⢠Temperature: 0.0 (deterministic decoding) for all API-based models. ⢠Max retries: 3, with exponential backoff (5 s, 10 s, 20 s). ⢠Timeout: 120â300 s depending on model (up to 600 s for local models). ⢠Max image dimension: 4000 px (images exceeding this are resized proportionally). ⢠Max output tokens: 8192 (where configurable). C.4 Post-Processing Model outputs are processed through a clean_markdown() function that strips Markdown code fences (e.g., âmarkdown ... â) from the response. No other normalization is applied to model outputs before evaluation. C.5 Human Baseline Protocol The human baseline is collected under conditions identical to model evaluation: ⢠Human participants receive the same handwritten document images as models. ⢠Single-pass recognition: no revision, no second attempt. ⢠No auxiliary resources: participants may not consult MLLM predictions, writer-provided transcripts, community discussions, or domain experts. ⢠Same target formats: Markdown for text, LaTeX for formulas, HTML for tables. ⢠Same evaluation metrics: Edit Distance, CDM, and TEDS are applied identically. This protocol yields a calibrated reference under model-comparable conditions rather than an absolute upper bound on human performance.