Paper deep dive
OmniHandwritingOCR: A Diagnostic Benchmark for Evaluating Multimodal LLMs in Handwritten OCR Scenarios
Zinuo Guo, Min Zhang, Bo Jiang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/20/2026, 4:48:48 AM
Summary
The paper introduces OmniHandwritingOCR, a diagnostic benchmark designed to evaluate Multimodal Large Language Models (MLLMs) and OCR systems on handwritten text recognition (HTR) and handwritten mathematical expression recognition (HMER). The benchmark comprises 77.57K labeled images from public datasets and newly collected student writings, covering six subtasks and twelve subsets. It emphasizes difficulty-stratified multi-line formulas to test robustness against structural complexity and visual grounding failures. The study evaluates thirteen open- and closed-source systems, revealing that current models struggle with faithful transcription, often hallucinating corrections or failing on complex layouts.
Entities (28)
Relation Signals (16)
Zinuo Guo → affiliatedwith → East China Normal University
confidence 95% · Zinuo Guo... East China Normal University Shanghai, China
Bo Jiang → affiliatedwith → East China Normal University
confidence 95% · Bo Jiang... East China Normal University Shanghai, China
Min Zhang → affiliatedwith → East China Normal University
confidence 95% · Min Zhang... East China Normal University Shanghai, China
OmniHandwritingOCR → coverstask → Handwritten Text Recognition
confidence 95% · It covers handwritten text recognition and handwritten mathematical expression recognition
OmniHandwritingOCR → coverstask → Handwritten Mathematical Expression Recognition
confidence 95% · It covers handwritten text recognition and handwritten mathematical expression recognition
OmniHandwritingOCR → evaluates → Multimodal Large Language Models
confidence 95% · We introduce OmniHandwritingOCR, a diagnostic benchmark for evaluating MLLMs and OCR systems on handwritten OCR.
OmniHandwritingOCR → publishedin → CIKM '26
confidence 95% · In Proceedings of 35th International ACM Conference on Information and Knowledge Management (CIKM ’26).
Qwen2.5-VL-72B → usedforfiltering → OmniHandwritingOCR
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multimodal large language models (MLLMs) are increasingly used as OCR systems in document and knowledge-processing pipelines, but their ability to faithfully read real handwriting remains underexplored. Existing OCR benchmarks focus largely on printed text or clean single-line inputs, leaving limited coverage of realistic handwritten OCR scenarios such as multilingual handwriting, writer errors, and structurally complex mathematical expressions. We introduce OmniHandwritingOCR, a diagnostic benchmark for evaluating MLLMs and OCR systems on handwritten OCR. It covers handwritten text recognition and handwritten mathematical expression recognition across six subtasks and twelve subsets, totaling 77.57K labeled images from public datasets and newly collected student writings. A key component is a difficulty-stratified multi-line formula corpus designed to test robustness under increasing structural complexity. We evaluate thirteen open- and closed-source systems with five complementary metrics under a unified protocol. Results show that current systems remain far from faithful transcription: performance drops sharply on complex multi-line formulas, model rankings vary across language and formula settings, and several generative models hallucinate plausible but visually unsupported corrections. OmniHandwritingOCR provides a challenging testbed for diagnosing language, content, structural, and visual-grounding failure modes of multimodal models in handwritten OCR scenarios.
Tags
Links
- Source: https://arxiv.org/abs/2608.18586v1
- Canonical: https://arxiv.org/abs/2608.18586v1
Trouble viewing inline? Open PDF directly →
Full Text
65,423 characters extracted from source content.
Expand or collapse full text
OmniHandwritingOCR: A Diagnostic Benchmark for Evaluating Multimodal LLMs in Handwritten OCR Scenarios Zinuo Guo 51275901001@stu.ecnu.edu.cn East China Normal University Shanghai, China Min Zhang ∗ mzhang@cs.ecnu.edu.cn East China Normal University Shanghai, China Bo Jiang bjiang@deit.ecnu.edu.cn East China Normal University Shanghai, China GNHK 515 (0.66%) IAM-line 11344 (14.62%) private Chinese compositions 874 (1.13%) HME100K 9754 (12.57%) MLHME 2010 (2.59%) private math problems (hard) 12899 (16.63%) C-OCR 100 (0.13%) private math problems (medium) 15488 (19.97%) private math problems (easy) 2135 (2.75%) CROHME 3332 (4.30%) CASIA-HWDB 11655 (15.02%) MLHME 7466 (9.62%) Figure 1: Overview of OmniHandwritingOCR. The benchmark focuses on two core tasks, handwritten text recognition (HTR) and handwritten mathematical expression recognition (HMER), covers six subtasks and twelve subsets with 77.57K images. ∗ Corresponding author & Project leader. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. CIKM ’26, Rome, ITALY © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-x-x-x-x/Y/M https://doi.org/10.1145/n.n Abstract Multimodal large language models (MLLMs) are increasingly used as OCR systems in document and knowledge-processing pipelines, but their ability to faithfully read real handwriting remains un- derexplored. Existing OCR benchmarks focus largely on printed text or clean single-line inputs, leaving limited coverage of realis- tic handwritten OCR scenarios such as multilingual handwriting, writer errors, and structurally complex mathematical expressions. We introduce OmniHandwritingOCR, a diagnostic benchmark for evaluating MLLMs and OCR systems on handwritten OCR. It arXiv:2608.18586v1 [cs.CV] 19 Aug 2026 CIKM ’26, November 7-11, 2026, Rome, ITALYZinuo Guo, Min Zhang, and Bo Jiang covers handwritten text recognition and handwritten mathemati- cal expression recognition across six subtasks and twelve subsets, totaling 77.57K labeled images from public datasets and newly col- lected student writings. A key component is a difficulty-stratified multi-line formula corpus designed to test robustness under in- creasing structural complexity. We evaluate thirteen open- and closed-source systems with five complementary metrics under a unified protocol. Results show that current systems remain far from faithful transcription: performance drops sharply on com- plex multi-line formulas, model rankings vary across language and formula settings, and several generative models hallucinate plausi- ble but visually unsupported corrections. OmniHandwritingOCR provides a challenging testbed for diagnosing language, content, structural, and visual-grounding failure modes of multimodal mod- els in handwritten OCR scenarios. The code is available at https: //github.com/ECNU-RAIL/OmniHandwritingOCR-CIKM2026. CCS Concepts • General and reference→ Evaluation. Keywords handwritten OCR, benchmark, multimodal large language models, evaluation, formula recognition, hallucination ACM Reference Format: Zinuo Guo, Min Zhang, and Bo Jiang. 2026. OmniHandwritingOCR: A Diagnostic Benchmark for Evaluating Multimodal LLMs in Handwritten OCR Scenarios. In Proceedings of 35th International ACM Conference on Information and Knowledge Management (CIKM ’26). ACM, New York, NY, USA, 12 pages. https://doi.org/10.1145/n.n 1 Introduction Optical character recognition (OCR) is a basic component of infor- mation acquisition and knowledge processing pipelines. In informa- tion and knowledge management settings, OCR outputs often be- come the input to indexing, search, retrieval-augmented generation, educational analytics, and document-level knowledge extraction systems. Modern systems are no longer limited to specialized OCR engines: multimodal large language models (MLLMs) [4,23,24,27] are now used to read documents, extract knowledge from images, and convert visual content into structured text. While these models have achieved strong performance on printed documents, their reliability on handwriting remains unclear. Real handwritten con- tent contains blurred strokes, corrections, irregular layouts, writer- specific styles, and incomplete or erroneous expressions. These properties are exactly where a generative model may stop recogniz- ing visual evidence and start producing plausible but unsupported text. Handwriting recognition is commonly studied through two re- lated tasks: handwritten text recognition (HTR) [11,15,17], which transcribes natural-language handwriting, and handwritten mathe- matical expression recognition (HMER) [1,2,48], which converts two-dimensional formula structures into markup such as L A T E X. Ex- isting resources have advanced both directions, but they leave im- portant evaluation gaps for current MLLM/OCR systems. Classical HTR datasets such as IAM [25] and RIMES [12] are largely monolin- gual and often cleaner than real educational writing. HMER datasets such as CROHME [42] and MathWriting [10] have driven progress on mathematical transcription, but are dominated by single-line expressions. As a result, existing evaluations do not sufficiently test multilingual handwriting, real student work, fact-preserving transcription, or complex multi-line formula structures. This limitation is not only a dataset coverage issue; it is an eval- uation issue. For LLM-based OCR systems, a useful benchmark should expose where models fail, how performance changes with structural complexity, and when outputs drift from visual evidence into plausible correction. Aggregate OCR scores can hide these failures, especially when a model appears fluent while omitting symbols, repairing student mistakes, or fabricating content that is not present in the image. Such errors directly affect downstream in- formation and knowledge management applications, where faithful transcription is often more important than plausible reconstruction. To address these needs, we introduce OmniHandwritingOCR, a diagnostic benchmark for evaluating handwritten OCR in MLLM/- OCR systems. It covers English text, Chinese text, single-line formu- las, and multi-line formulas stratified into easy, medium, and hard subsets, totaling 77.57K image-label pairs from public and newly collected sources. Its diagnostic design supports analysis along four axes: language, content type, structural complexity, and visual grounding. This allows the benchmark to report not only which system ranks highest overall, but also where and why a model fails under multilingual, long-context, and structurally complex handwritten inputs. Our contributions are as follows: • We construct a large-scale handwritten OCR benchmark that integrates HTR and HMER, covering six subtasks and twelve subsets with 77.57K labeled images. •We introduce a difficulty-stratified multi-line formula eval- uation setting that targets complex real student work and enables analysis of model degradation as structural complex- ity increases. •We define a unified evaluation setting with task-specific nor- malization and tokenization rules, enabling fair comparison across general-purpose MLLMs and specialized OCR models. •We benchmark thirteen open- and closed-source systems and identify diagnostic failure modes, including language- specific performance shifts, sensitivity to formula structure, and hallucinated corrections on ambiguous handwriting. 2 Related Work We review prior work from the perspective of handwritten recogni- tion and benchmark coverage, focusing on handwritten text recog- nition (HTR), handwritten mathematical expression recognition (HMER), and the gap between existing resources and realistic hand- written OCR scenarios for current MLLM/OCR systems. 2.1 Handwritten Text Recognition Handwritten text recognition (HTR) has long been studied as a core document analysis problem. Earlier systems relied on statistical sequence modeling, including HMM-based recognizers [6], while modern systems commonly use CNN-RNN-CTC architectures [33], multidimensional recurrent models [11], attention-based recogniz- ers [3], and Transformer-based models such as TrOCR [20]. These OmniHandwritingOCR: A Diagnostic Benchmark for Evaluating Multimodal LLMs in Handwritten OCR ScenariosCIKM ’26, November 7-11, 2026, Rome, ITALY Data Collection and AnnotationData Filtering and Classifying 6K answer sheets 50+ experts 86K annotated images 7 mainstream public datasets 160K samples public data private data contain 30K annotated solution questions Qwen2.5-VL-72B abandon retain easy, medium, hard 86K annotated images 46K samples classify into 6 subtasks Omni Handwriting OCR Figure 2: Construction pipeline of OmniHandwritingOCR, including data collection, filtering, difficulty stratification, annota- tion, and expert verification. methods have substantially improved line-level and document-level transcription, and they remain important baselines for understand- ing how visual encoders and sequence decoders handle handwritten variation. However, real handwritten text still poses challenges be- yond isolated character recognition, including inconsistent spacing, writer-specific abbreviations, touching strokes, insertions, deletions, and layout changes across lines. Progress in HTR has been driven by public benchmarks such as IAM [25], RIMES [12], READ-BAD [13], GNHK [19], CASIA- HWDB [22], and SCUT-HCCDoc [45]. These datasets cover impor- tant scenarios including English handwriting, French mail, histori- cal documents, in-the-wild handwriting, and Chinese handwritten text. However, they are often organized around specific languages, sources, or document types, and many samples are cleaner than real educational handwriting. They also do not directly evaluate how general-purpose MLLMs behave when handwriting contains writer errors, corrections, irregular layouts, or mixed textual and mathematical content. Recent OCR and document benchmarks for MLLMs, such as OmniDocBench [29], OCRBenchv2 [9], and C-OCR [43], broaden evaluation to layout-rich, multilingual, and document-level tasks. Nevertheless, handwriting remains only one component within these broader resources. OmniHandwritingOCR follows the HTR tradition but extends it toward MLLM-era evaluation by combining English and Chinese handwriting with fact-based annotations and unified scoring, enabling direct diagnosis of faithful transcription under realistic handwritten conditions. 2.2 Handwritten Mathematical Expression Recognition Handwritten mathematical expression recognition (HMER) is more structurally demanding than ordinary text recognition because a system must identify symbols and recover two-dimensional rela- tions. Classical and neural approaches have explored grammar- based parsing [1], online trajectory modeling [2], sequence de- coders [46], tree-structured decoders [47], graph modeling [32], and structure-aware Transformers such as TAMER [50]. These methods show that formula recognition requires explicit or implicit modeling of spatial relations, not only character-level transcription. In this setting, small recognition errors such as missing braces, misplaced superscripts, or incorrect fraction scopes can substan- tially change the recovered expression, making structural fidelity as important as symbol accuracy. Benchmarks such as CROHME [42], HME100K [44], MathWrit- ing [10], UniMER [37], and MLHME [34] are central to HMER re- search. They provide standardized evaluation data and have enabled progress on symbol recognition, L A T E X sequence generation, and expression-level parsing. However, much of the existing evaluation is still dominated by single-line expressions or relatively controlled formula layouts. Even when multi-line formulas are included, they usually do not fully capture the long, noisy, and correction-heavy derivations found in real student answer sheets. This gap is important for current MLLM/OCR systems. A model may recognize isolated symbols or short formulas while failing to preserve alignment, multi-step dependencies, cases, fractions, superscripts, or crossed-out content in longer handwritten deriva- tions. OmniHandwritingOCR therefore integrates HTR and HMER in a single benchmark and places special emphasis on difficulty- stratified multi-line formulas. By combining public formula datasets with newly collected student answer sheets, it evaluates not only formula transcription accuracy, but also robustness to the struc- tural complexity that appears in realistic handwritten mathematical work. Compared with prior HTR or HMER resources, our focus is not to replace task-specific benchmarks, but to provide a unified stress test for systems that are increasingly deployed as general handwritten OCR engines. This is particularly relevant for MLLMs, whose outputs may be influenced by language priors, mathematical priors, and instruction-following behavior. Evaluating text and formula handwriting under the same protocol makes these cross- task effects visible and helps distinguish genuine visual recognition from plausible reconstruction. 3 OmniHandwritingOCR To evaluate handwriting recognition capabilities beyond aggregate OCR scores, we design OmniHandwritingOCR around three re- quirements: broad task coverage, difficulty-aware diagnosis, and standardized scoring. The benchmark contains more than 77K sam- ples, spans English text, Chinese text, and mathematical formulas, and systematically stratifies multi-line formula content by difficulty. This dual focus on breadth and depth enables fine-grained analysis CIKM ’26, November 7-11, 2026, Rome, ITALYZinuo Guo, Min Zhang, and Bo Jiang English Text Chinese Text Single-line Formula Multi-line Formula (easy) Multi-line Formula (medium) Multi-line Formula (hard) Figure 3: Representative examples from each subtask, showing the diversity of handwriting styles and the progressive difficulty of formula structures. of model generalization across languages, layouts, and structural complexity. 3.1 Design Principles OmniHandwritingOCR is designed to support diagnostic evalua- tion rather than only leaderboard ranking. The first principle is task diversity. Real handwritten OCR applications rarely involve a single clean text line. They may contain English notes, Chinese compositions, mathematical expressions, long derivations, correc- tions, and mixed natural-language/formula content. We therefore combine HTR and HMER in one benchmark so that models can be compared under a shared protocol while still exposing task-specific strengths and weaknesses. The second principle is difficulty-aware structure. For formula recognition, the central challenge is not only identifying symbols, but also preserving two-dimensional relations such as fractions, superscripts, cases, and aligned multi-step derivations. A bench- mark dominated by single-line expressions can overestimate model readiness for real educational or scientific handwriting. OmniHand- writingOCR therefore separates single-line formulas from multi- line formulas and further stratifies multi-line samples by difficulty, enabling controlled analysis of how models degrade as visual and symbolic structure becomes more complex. The third principle is faithful transcription. In many knowledge- processing pipelines, OCR output is treated as evidence for index- ing, retrieval, or downstream reasoning. A model that corrects a student’s mistake or fills in a missing derivation may produce a plausible answer, but it is no longer faithfully transcribing the im- age. Our fact-based annotation policy and unified scoring protocol are designed to penalize such visually unsupported corrections. This makes the benchmark especially suited to evaluating genera- tive OCR systems, where hallucinated correction can be hidden by fluent output. OmniHandwritingOCR: A Diagnostic Benchmark for Evaluating Multimodal LLMs in Handwritten OCR ScenariosCIKM ’26, November 7-11, 2026, Rome, ITALY Ground Truth: 第一种：$1\×4\×500 + 2\×3\×500+3\×2\×800+ 3\×2\×800+5\×2\×800 = 20200$(元） 第二种：$(6 + 4 + 3 + 6 + 10)\×600 = 1540$(元) 答：选第二种最合适 InternVL3-78B: 方案一 : 1×4×300 + 2×3×500 + 3×2×800 = 3×2×800 + 5×2×800 = 20200 (元) 方案二 : (6 + 4 + 3 + 6 + 10) ×600 = 15400 (元) 答:选方案二中最合适 Ground Truth: 解：(1)：双曲线$C:\ x^2a^2-\ y^2b^2 = 1 (a>0,b>0)$ 且离心率为$\ 2$，$A_0(2,\ 3)$是双曲线$C$上 一点, $C=\ x^22^2-\ y^2\ 3^2 = 1$ $C=\ x^24-\ y^29=1$ Qwen2.5-VL-72B: 解：(1) 双曲线 $C:\ x^2a^2-\ y^2b^2 = 1(a>0,b>0)$. 已知离心率为 $\ 2$，$A_0 (2, \ 3)$ 是双曲线 $C$ 上一点. $C = \ x^22^2- \ y^2(\ 3)^2 = 1$ $C = \ x^24-\ y^23 = 1$ Figure 4: Examples of fact-based annotations. Ground-truth labels preserve visible writer mistakes, while Qwen2.5-VL- 72B and InternVL3-78B incorrectly produce corrected but visually unsupported transcriptions. Table 1: Detailed composition of the dataset. SubtaskSourceSamples Total English Text GNHK515 11,859 IAM-line11,344 Chinese Text private874 12,529 CASIA-HWDB11,655 Single-line Formula CROHME3,332 15,096HME100K9,754 MLHME2,010 Multi-line Formula (easy) MLHME7,466 9,601 private2,135 Multi-line Formula (medium) private15,48815,488 Multi-line Formula (hard) C-OCR100 12,999 private12,899 Total77,572 3.2 Benchmark Construction In this section, we provide a detailed overview of our construction pipeline for OmniHandwritingOCR, as shown in Figure 2. The pipeline includes dataset collection to ensure breadth and novelty, data filtering and difficulty stratification to ensure evaluative rigor, and finally, expert annotation to ensure ground-truth accuracy. 3.2.1 Dataset Collection. Our data collection strategy employs a two-pronged approach. First, we aggregated key public datasets to establish a broad foundation. We then supplemented this with a substantial, newly collected private dataset to target specific and challenging domains. Public Data Aggregation. To establish a comprehensive founda- tion, we gathered samples from influential public datasets, including IAM-line [25] and GNHK [19] for English text, CASIA-HWDB [22] for Chinese text, and CROHME [42], HME100K [44], MLHME38K [34], and C-OCR [43] for mathematical formulas. This initial step provided a diverse pool of approximately 160K samples, covering a wide range of established tasks. Private Data Collection. Our private data is collected specif- ically to address the most critical gaps in existing resources. We sourced it from two key areas: (1) 874 authentic student Chinese lan- guage compositions, which fill a crucial void by providing complex, real-world multi-line samples, and (2) our primary contribution: a first-of-its-kind dataset derived from over 6,000 authentic student mathematics answer sheets. From these, we segmented and anno- tated 86K images. This unique corpus captures the natural, often chaotic, and structurally complex nature of handwritten problem- solving, a scenario largely absent from existing datasets that are dominated by clean, single-line expressions. 3.2.2 Data Filtering and Difficulty Stratification. A primary goal was to create a benchmark that actively challenges state-of-the-art models, rather than just measuring performance on solved problems. To this end, we implemented a rigorous, two-part filtering and stratification process. First, for the public data, we performed a challenge-oriented filtering to increase difficulty. We leveraged a powerful vision- language model, Qwen2.5-VL-72B, to recognize the initial 160K samples. By calculating the normalized edit distance (NED) between the model’s prediction and the ground truth, we were able to sys- tematically identify and discard samples that the model found easy. This deliberate selection of poorly performing samples allowed us to distill a final set of 46K high-quality, challenging samples that push the boundaries of current model capabilities. Second, for the private math data, we first focused the dataset by filtering the 86K segmented images to retain only the core solution sections, resulting in 30K high-quality samples of multi-line and difficult formulas. Furthermore, to enable detailed model evaluation, we designed a quantitative difficulty scoring system. This system scores each sample with three factors: the length of the ground truth (weight 0.2, representing information volume), the number of complex L A T E X commands (weight 0.3, representing structural complexity), and the recognition NED from GPT-4o (weight 0.5, rep- resenting perceptual difficulty for a strong vision-language model). We use this score only to stratify evaluation subsets, not as a train- ing signal or as an additional metric. Because one component is model-informed, we validate the resulting split with model-agnostic structural statistics below. Based on this composite score, we clas- sify the samples into easy, medium, and hard categories, forming the hierarchical structure that supports the diagnostic experiments in Section 4. To reduce the risk that the difficulty split is merely tied to one model’s behavior, we also inspect model-agnostic label statistics. As shown in Table 2, the three private multi-line formula subsets exhibit monotonic increases in ground-truth length, L A T E X command density, and number of non-empty lines. This validates that the split reflects observable structural complexity in addition to model recognition difficulty. 3.2.3Expert Annotation and Verification. Precise and high-quality annotation is critical for a fair and research-worthy benchmark. Our annotation strategy was tailored to the data’s origin to ensure con- sistency and accuracy. For all public data, we retained their original, CIKM ’26, November 7-11, 2026, Rome, ITALYZinuo Guo, Min Zhang, and Bo Jiang Table 2: Model-agnostic statistics of private multi-line for- mula subsets. Len. denotes average ground-truth characters, Cmd. denotes average L A T E X commands, and Lines denotes average non-empty text lines. SubsetLen. Cmd. Lines ml-easy156.94.13.7 ml-medium213.07.67.1 ml-hard488.825.713.0 widely-used annotations to maintain consistency for comparison with existing research and also honor the significant community effort invested in creating these established ground truths. The annotation of our private data, however, demanded a sub- stantial effort. We assembled a dedicated team of over 50 domain experts for an intensive two-month project. The final benchmark contains 31,396 private samples, including 874 Chinese composi- tion samples and 30,522 private mathematical-expression samples. The private mathematical-expression data were derived from over 6,000 authentic student answer sheets, and each original answer sheet underwent two rounds of human checking before its seg- mented samples were finalized. In the first round, a human expert verified and corrected the model-generated pre-annotations. The human-refined results then served as higher-quality baselines for a second round of expert review and refinement. Ambiguous cases were resolved by additional inspection rather than majority voting, because many errors involve fine visual details such as superscripts, omitted strokes, or crossed-out content. This iterative process was designed to systematically minimize errors and ensure the final ground truth achieved the highest possible level of accuracy. A cornerstone of this process was our fact-based annotation principle: maintaining strict fidelity to the writer’s original hand- writing. We intentionally retained common writer errors, such as misspellings or miscopied numbers, rather than correcting them. Furthermore, any content crossed out by the writer was omitted entirely. This philosophy is crucial because it forces models to learn robust visual pattern recognition instead of relying on contextual guessing or hallucinating corrections, as shown in Figure 4. By creating a ground truth that reflects what is visually present, our benchmark provides a more accurate tool for pinpointing specific model weaknesses and driving future improvements in genuine recognition capabilities. Privacy and ethical handling. The private portion consists of educational handwriting samples. Before benchmark construc- tion, personally identifying metadata was removed, and images were segmented to retain only task-relevant handwritten content. The released benchmark uses anonymized identifiers and excludes information that is not needed for recognition evaluation. 3.3 Dataset Statistics OmniHandwritingOCR is a large-scale collection totaling 77,572 image-label pairs across 6 different subtasks. The detailed compo- sition, as shown in Table 1, is balanced to cover a wide range of handwritten recognition tasks, from simple text lines to complex, multi-line mathematical expressions. To elaborate, the English text (en-text) combines 515 images from GNHK and 11,344 from IAM-line. The Chinese text (zh-text) includes 11,655 images from CASIA-HWDB, significantly supple- mented by our 874 privately collected multi-line Chinese compo- sitions. A core contribution of our work lies in the mathematical formula portion. The benchmark includes 15,096 single-line (sl) formulas sourced from CROHME (3,332), HME100K (9,754), and MLHME (2,010). More critically, it features our large, unique corpus of multi-line (ml) formulas, which are stratified by our difficulty scoring system into 9,601 ml-easy, 15,488 ml-medium, and 12,999 ml-hard samples. As shown in Table 3, OmniHandwritingOCR demonstrates clear advantages in both task comprehensiveness and structural com- plexity. It uniquely combines handwritten text with handwritten mathematical expressions, allowing it to more closely simulate real-world scenarios involving mixed content. Furthermore, our benchmark contains a substantial number of handwritten multi-line formula samples, 38,088 in total, which is significantly more than the other benchmarks compared. This provides a more challenging platform for evaluating a model’s ability to process complex spatial layouts. 4 Experiments In this section, we evaluate state-of-the-art methods on OmniHand- writingOCR under a unified offline protocol. The goal is not only to rank models, but also to diagnose how their behavior changes across language, handwriting type, and formula complexity. 4.1 Evaluation Protocol Baselines. We evaluate 13 methods in total, including 7 general- purpose MLLMs and 6 specialized OCR models. To reflect current practice, the benchmark includes both closed-source systems and newly released open-source models. The MLLMs are GPT-4o [28], Qwen2.5-VL-72B [5], InternVL3-78B [49], Gemma-3-27B-it [35], DeepSeek-VL2-27B [41], Kimi-VL-A3B-Instruct [36], and Qwen3- VL-8B [31]. The OCR models are MonkeyOCR-pro-1.2B [21], MinerU2.5- 1.2B [38], Nanonets-OCR2-3B [26], GOT-OCR2 [39], PaddleOCR- VL [8], and DeepSeek-OCR [40]. Task format. All models are evaluated in a zero-shot transcrip- tion setting. Each input consists of a single image and a task-neutral instruction equivalent to: “convert the text and formulas in the im- age into Markdown, and do not include any other content.” This prompt is intentionally short so that the evaluation measures vi- sual transcription rather than task-specific reasoning or prompt engineering. No retrieval, task-specific demonstrations, or dataset- specific context is provided. Open-source models are served through OpenAI-compatible local inference services when possible, while closed-source models are evaluated through their official APIs. We use deterministic decoding settings when supported by the model interface. Normalization and tokenization. We apply the same post- processing to every model output, following the released evaluation script. Markdown code fences and explicit format wrappers such asplaintext,markdown, andlatexare removed before scoring. To prevent runaway generations from dominating edit-distance metrics, single-line formula outputs are capped at 128 characters OmniHandwritingOCR: A Diagnostic Benchmark for Evaluating Multimodal LLMs in Handwritten OCR ScenariosCIKM ’26, November 7-11, 2026, Rome, ITALY Table 3: Comparison of different handwriting-related benchmarks, where HTR denotes Handwritten Text Recognition, HMER denotes Handwritten Mathematical Expression Recognition, and MLF denotes Handwritten Multi-Line Formula. BenchmarkTypeTasksStructure Num AllMLF RIMES-2011-line [12]HTR1single-line only12,104- READ-2016 [14]HTR1multi-line only30,000- SCUT-HCCDoc [45]HTR5single-line & multi-line12,253- CROHME2023 [42]HMER1single-line only13,2790 MLHME-38K [34]HMER2single-line & multi-line38,0009,931 Mathwriting [10]HMER1single-line only649,0000 UniMER [37]HMER4single-line & multi-line1,085,5480 OursHTR+HMER6single-line & multi-line77,57238,088 Table 4: Experiments (%) on the entire OmniHandwritingOCR. The highest is bold, and the second highest is underlined. MethodsBLEU-4↑ F1 Score↑ 1-NED↑ Overall↑CER↓ WER↓ Overall↓ Multimodal large language models (MLLMs) GPT-4o39.5367.6659.3255.5048.5350.0349.28 Qwen2.5-VL-72B56.4079.9872.9169.7638.8836.9137.90 InternVL3-78B53.7678.2172.1668.0432.1934.7233.46 Gemma-3-27B-it31.8460.0850.9747.6365.8267.8366.83 DeepSeek-VL2-27B39.5763.0654.9052.5176.9163.4970.20 Kimi-VL-A3B-Instruct58.7578.9073.5870.4138.2045.2841.74 Qwen3-VL-8B59.2481.6075.6372.16 30.94 32.9431.94 Optical character recognition (OCR) models MonkeyOCR-pro-1.2B49.3770.6066.5262.1651.5454.1152.83 MinerU2.5-1.2B8.4825.1221.3318.31125.59124.65125.12 Nanonets-OCR2-3B53.8474.1368.4265.4642.27 42.7742.52 GOT-OCR229.0354.1948.3243.8584.1977.6580.92 PaddleOCR-VL50.2169.6162.7360.8571.1472.3871.76 DeepSeek-OCR47.5171.9664.2761.2573.8469.9471.89 and other outputs are capped at 1024 characters before metric com- putation. For English text, line breaks are normalized to spaces and outputs are tokenized at the word level. For Chinese text, line breaks are removed and Jieba tokenization is used for token-level metrics. For public formula subsets, L A T E X delimiters and non-semantic white- space are normalized and tokens are extracted with a command- aware regular expression. For private mixed Chinese-formula sub- sets, we use a hybrid tokenizer that separates Chinese spans from L A T E X-style commands, numbers, operators, and punctuation. No model-specific correction, manual cleanup, or outlier removal is applied during final scoring. Evaluation safeguards. The protocol is intentionally conser- vative. We use a single task-neutral instruction for all systems to avoid tuning prompts for particular model families or subsets. We also avoid semantic correction during scoring: an output that re- pairs a writer error, changes an incorrect formula into a correct one, or adds a missing reasoning step is penalized if the content is not visually supported by the image. The output-length caps do not change ordinary transcriptions, but they prevent pathological generations from overwhelming edit-distance metrics and mak- ing a small number of failures dominate aggregate scores. These safeguards make the evaluation closer to an offline benchmark of visual transcription than to a test of problem-solving or answer reconstruction. 4.2 Metrics We employ five complementary metrics and report all values as percentages. BLEU-4 [30] measures 4-gram overlap between the model output and the ground truth. Token-level F1 [7] balances precision and recall, rewarding outputs that include the correct tokens without excessive insertion. CER and WER [16] measure character-level and token-level edit rates, respectively, so lower values indicate fewer transcription errors. Finally, 1-NED [18] con- verts normalized Levenshtein distance into an accuracy-style score, with 1 indicating an exact match. We treat 1-NED as the primary metric because it is robust across text and formula settings while CIKM ’26, November 7-11, 2026, Rome, ITALYZinuo Guo, Min Zhang, and Bo Jiang Table 5: Experiments (%) on the English (en-text) and Chinese text (zh-text). The highest is bold, and the second highest is underlined. Methods BLEU-4↑F1 Score↑1-NED↑CER↓WER↓ en-textzh-texten-textzh-texten-textzh-texten-textzh-texten-textzh-text Multimodal large language models (MLLMs) GPT-4o54.0333.5177.4458.4881.2960.66 20.0543.9031.9351.84 Qwen2.5-VL-72B58.5771.5181.8488.3984.6390.7116.01 10.42 28.21 15.16 InternVL3-78B51.2656.5976.9877.4281.1479.91 20.3622.5334.6929.08 Gemma-3-27B-it40.0713.9369.3633.0674.8629.2826.1295.1842.70101.30 DeepSeek-VL2-27B47.8322.9972.2243.7276.7240.71 27.89133.8542.7489.61 Kimi-VL-A3B-Instruct43.6557.5370.7776.7370.7776.1537.9640.3463.2260.03 Qwen3-VL-8B58.3772.2481.7888.72 84.76 90.8516.4411.8928.4317.94 Optical character recognition (OCR) models MonkeyOCR-pro-1.2B53.4668.0977.8281.8382.4179.6623.6478.6741.7869.15 MinerU2.5-1.2B11.6214.4625.1326.6532.2325.87 93.43150.71134.46112.94 Nanonets-OCR2-3B59.7678.1781.4589.6084.9688.2519.41 22.73 36.79 23.62 GOT-OCR221.0524.4347.4748.0559.8850.4543.1762.4070.2669.07 PaddleOCR-VL46.65 81.3968.98 91.6573.66 89.0547.4832.572.0539.64 DeepSeek-OCR45.7566.7770.1479.4877.3376.9539.8376.0664.0959.14 Table 6: Experiments (%) on single-line (sl) and easy multi-line (ml-easy). The highest is bold, and the second highest is underlined. Methods BLEU-4↑F1 Score↑1-NED↑CER↓WER↓ slml-easyslml-easyslml-easyslml-easyslml-easy Multimodal large language models (MLLMs) GPT-4o24.5161.0862.2681.6851.9871.1779.2429.3774.5026.35 Qwen2.5-VL-72B40.0964.2472.9882.8463.1371.7367.0735.5357.5330.13 InternVL3-78B56.6767.0084.0885.1380.7075.8127.0426.0430.3423.49 Gemma-3-27B-it19.3652.0353.7475.9645.8364.16104.3140.00108.9535.03 DeepSeek-VL2-27B67.64 42.9687.8066.2086.0649.7218.5078.3119.7461.00 Kimi-VL-A3B-Instruct 78.40 84.98 93.33 91.99 93.60 88.82 10.91 21.93 14.20 21.07 Qwen3-VL-8B52.7770.6379.8886.9274.2079.5740.2724.9538.7423.53 Optical character recognition (OCR) models MonkeyOCR-pro-1.2B 69.0956.58 88.9071.18 87.56 74.66 16.06 30.39 16.9445.17 MinerU2.5-1.2B5.652.6324.6216.2324.4211.63132.67123.10107.29108.87 Nanonets-OCR2-3B48.2657.5673.7374.4968.8064.8047.0842.1248.6439.13 GOT-OCR256.2528.5178.5156.4674.4039.9557.2583.0252.4170.45 PaddleOCR-VL38.7549.3953.4665.7750.0055.88116.7774.30108.5765.87 DeepSeek-OCR36.97 58.8866.01 79.6555.5468.59154.8138.13118.08 35.50 still penalizing substitutions, deletions, and hallucinated insertions. In tables with aggregate columns, Overall↑is the arithmetic mean of BLEU-4, F1, and 1-NED, while Overall↓is the arithmetic mean of CER and WER. 4.3 Main results Table 4 and Figure 5 summarize overall performance. Qwen3-VL-8B obtains the strongest aggregate score among all evaluated systems, with 59.24 BLEU-4, 81.60 F1, 75.63 1-NED, and the lowest aver- age error rate among the MLLMs. However, the aggregate ranking also shows that handwritten OCR is far from solved: even the best model leaves a large gap to exact transcription, and several spe- cialized OCR systems remain competitive on particular metrics. Among OCR models, Nanonets-OCR2-3B achieves the best over- all OCR-model score and lower CER/WER than other specialized OmniHandwritingOCR: A Diagnostic Benchmark for Evaluating Multimodal LLMs in Handwritten OCR ScenariosCIKM ’26, November 7-11, 2026, Rome, ITALY Table 7: Experiments (%) on the medium multi-line formula (ml-medium) and hard multi-line formula (ml-hard). Methods BLEU-4↑F1 Score↑1-NED↑CER↓WER↓ ml-mediumml-hardml-mediumml-hardml-mediumml-hardml-mediumml-hardml-mediumml-hard Multimodal large language models (MLLMs) GPT-4o46.8417.1873.4352.6861.2029.6043.0575.5843.0672.52 Qwen2.5-VL-72B59.98 43.9880.4273.3971.4755.8149.3254.9142.0048.42 InternVL3-78B57.5333.5279.3566.3269.8145.5936.39 60.8034.8355.89 Gemma-3-27B-it43.6122.0370.3258.0557.3234.3856.1173.2149.9569.05 DeepSeek-VL2-27B37.9518.0762.4545.9948.2527.94109.0493.8781.9185.92 Kimi-VL-A3B-Instruct56.2931.6378.1462.4668.6743.4848.8669.2046.7066.47 Qwen3-VL-8B61.6539.77 81.8270.4573.8350.5434.9857.0834.4554.52 Optical character recognition (OCR) models MonkeyOCR-pro-1.2B26.7822.3351.8953.8440.5933.2383.5976.9177.7873.81 MinerU2.5-1.2B9.986.5329.2828.8317.8216.23122.57131.06163.02120.28 Nanonets-OCR2-3B48.5330.88 72.6862.8062.0441.6754.7167.5146.4462.44 GOT-OCR229.1514.9055.1339.5341.1724.26145.46113.82110.58103.14 PaddleOCR-VL49.68 35.0169.7162.5158.2241.7894.1683.5172.5876.52 DeepSeek-OCR52.9430.2175.9161.5864.9642.2453.4970.7148.9165.91 GPT-4o Qwen2.5-VL-72B InternVL3-78B Gemma-3-27B-it Deepseek-VL2-27B Kimi-VL-A3B-Instruct Qwen3-VL-8B MonkeyOCR-pro-1.2B MinerU2.5-1.2B Nanonets-OCR2-3B GOT-OCR2 PaddleOCR-VL Deepseek-OCR 0 15 30 45 60 75 90 BLEU-4F1 Score1-NED GPT-4o Qwen2.5-VL-72B InternVL3-78B Gemma-3-27B-it Deepseek-VL2-27B Kimi-VL-A3B-Instruct Qwen3-VL-8B MonkeyOCR-pro-1.2B MinerU2.5-1.2B Nanonets-OCR2-3B GOT-OCR2 PaddleOCR-VL Deepseek-OCR 0 15 30 45 60 75 90 105 120 135 CERWER GPT-4o Qwen2.5-VL-72B InternVL3-78B Gemma-3-27B-it Deepseek-VL2-27B Kimi-VL-A3B-Instruct Qwen3-VL-8B MonkeyOCR-pro-1.2B MinerU2.5-1.2B Nanonets-OCR2-3B GOT-OCR2 PaddleOCR-VL Deepseek-OCR 0 15 30 45 60 75 90 BLEU-4F1 Score1-NED GPT-4o Qwen2.5-VL-72B InternVL3-78B Gemma-3-27B-it Deepseek-VL2-27B Kimi-VL-A3B-Instruct Qwen3-VL-8B MonkeyOCR-pro-1.2B MinerU2.5-1.2B Nanonets-OCR2-3B GOT-OCR2 PaddleOCR-VL Deepseek-OCR 0 15 30 45 60 75 90 105 120 135 CERWER Figure 5: Overall benchmark performance (%) under BLEU-4, F1, 1-NED, CER, and WER. OCR baselines, indicating that domain-specific OCR training still provides advantages for precise transcription. A closer comparison across model groups shows that overall handwritten OCR performance is not determined by model size alone. Qwen3-VL-8B outperforms larger MLLMs such as Qwen2.5- VL-72B and InternVL3-78B in the aggregate table, while Kimi-VL- A3B-Instruct later becomes strongest on several formula subsets. The OCR baselines also show substantial variation: Nanonets-OCR2- 3B and MonkeyOCR-pro-1.2B remain competitive, whereas some document-oriented OCR models degrade sharply on handwritten formula content. These results suggest that training data, visual resolution handling, and formula-aware generation matter as much as general multimodal capacity for handwritten OCR. The per-language results in Table 5 show that model behavior is strongly language dependent. The Qwen series is consistently strong on Chinese handwriting, with Qwen2.5-VL-72B reaching 10.42 CER and 15.16 WER on zh-text. In contrast, Nanonets-OCR2- 3B leads most English-text metrics among OCR models, while PaddleOCR-VL achieves the highest F1 and 1-NED on Chinese text within the OCR group. These results indicate that a single over- all handwritten OCR score can obscure language-specific strengths and weaknesses. Formula recognition further changes the ranking, as shown in Tables 6 and 7. Kimi-VL-A3B-Instruct is the strongest model on single-line and easy multi-line formulas, reaching 93.33 F1 and 10.91 CER on single-line formulas. As the multi-line formula subset becomes harder, the leading model changes: Qwen3-VL-8B leads on ml-medium, while Qwen2.5-VL-72B is more robust on ml-hard. This rank instability confirms that OmniHandwritingOCR evaluates more than generic OCR ability; it exposes task-specific robustness under changing layout and structural complexity. 4.4 Analysis and Findings Structural complexity is the dominant stressor. All model families degrade as the benchmark moves from easy to hard multi- line formulas. For example, Qwen3-VL-8B drops from 86.92 F1 on ml-easy to 70.45 F1 on ml-hard, while its CER worsens from 24.95 to 57.08. This trend is consistent with the model-agnostic statistics in Table 2: average label length increases from 156.9 to 488.8 characters, average L A T E X command count increases from 4.1 to 25.7, and average non-empty lines increase from 3.7 to 13.0. The degradation therefore cannot be explained only by harder handwriting styles. Hard samples require models to maintain long symbolic dependencies and preserve two-dimensional relationships such as fractions, superscripts, cases, and aligned derivations. The CIKM ’26, November 7-11, 2026, Rome, ITALYZinuo Guo, Min Zhang, and Bo Jiang results suggest that current MLLM/OCR systems still struggle to convert complex handwritten spatial structures into faithful linear markup. Model rankings are task-conditioned. The strongest over- all model is not uniformly strongest across all handwritten OCR scenarios. Qwen3-VL-8B leads the aggregate score, but Qwen2.5- VL-72B achieves the best Chinese-text CER/WER, Kimi-VL-A3B- Instruct dominates single-line and easy multi-line formulas, and Nanonets-OCR2-3B is highly competitive on English handwriting and OCR-model aggregates. This rank instability is important for evaluation: a single leaderboard score can hide whether a model is robust to language changes, formula structure, or long multi-line derivations. OmniHandwritingOCR is therefore designed to report both aggregate and task-conditioned performance. General-purpose and specialized models fail differently. General-purpose MLLMs tend to be stronger on broad cross-task aggregation, while specialized OCR models show sharper strengths on particular data regimes. Specialized OCR systems often pro- duce more compact transcriptions, but may fail on mixed natural- language and formula content or on long reasoning-style deriva- tions. Conversely, MLLMs can leverage language priors and broad visual understanding, but those same priors can introduce unsup- ported corrections when the image is ambiguous. This contrast suggests that handwritten OCR evaluation should measure not only recognition accuracy, but also whether the output remains visually grounded. Generative recognition introduces hallucinated correction. The fact-based labels allow us to observe a failure mode that is dif- ficult to measure in ordinary OCR datasets: models may correct, complete, or fabricate content rather than transcribe what is visually present. We observe four recurring error patterns: correcting writer mistakes, inserting plausible intermediate steps, dropping visually present but hard-to-read symbols, and drifting into Markdown or explanatory formatting. This behavior is reflected quantitatively by error rates above 100 in some settings, such as DeepSeek-VL2-27B’s 133.85 CER on Chinese handwriting and DeepSeek-OCR’s 154.81 CER on single-line formulas. Such values mean that insertions dom- inate the edit distance, which is consistent with over-generation or hallucinated transcription. Figure 4 further illustrates cases where models correct students’ original mistakes, producing semantically plausible outputs that are wrong under a faithful-recognition crite- rion. Implications for knowledge processing. These findings are important beyond OCR accuracy itself. In retrieval, indexing, educa- tional analytics, and document understanding systems, recognized text is often stored and reused as if it were a faithful representation of the source image. Deletions can remove searchable evidence, substitutions can corrupt formulas or named entities, and halluci- nated corrections can introduce facts that were never written. The benchmark therefore highlights a practical requirement for hand- written OCR in knowledge pipelines: systems should preserve visual evidence even when the image contains mistakes or incomplete reasoning. This also explains why aggregate scores are insufficient. A model that performs well on clean text may still be unreliable for long mathematical derivations or mixed handwriting, where downstream systems need traceable transcription rather than fluent reconstruction. 5 Data Release, Ethics, and Limitations Data release and reproducibility. OmniHandwritingOCR is de- signed as an offline benchmark with deterministic scoring once model outputs are generated. We will publicly release the bench- mark images, labels, and split manifests, with anonymized iden- tifiers for newly collected educational data. We will also release the evaluation scripts that implement the normalization, tokeniza- tion, output-length caps, and metric computation described above. Public-source subsets follow their original dataset licenses, and all released private samples are anonymized to remove personally identifying information. Ethics. The private handwriting samples come from educational scenarios and are used only for recognition evaluation. Person- ally identifying metadata is removed, and the benchmark does not require demographic labels. Our fact-based annotation policy inten- tionally preserves visible writer errors; this design is important for evaluating faithful transcription, but it should not be interpreted as judging the writer’s ability or correctness. Limitations. The benchmark focuses on offline image-based handwriting and does not include online pen trajectories. Its for- mula scoring is based on L A T E X-style textual comparison, which can penalize semantically equivalent expressions written in different markup forms. The difficulty score includes a model-performance component, so it should be viewed as an evaluation stratification tool rather than an intrinsic human difficulty measure; the model- agnostic statistics in Table 2 are included to show that the splits also reflect observable structural complexity. In addition, some public subsets may have appeared in the pretraining data of large multimodal models, which is a common concern for public bench- mark evaluation. The newly collected private educational samples therefore provide a less exposed stress test for current systems. Future extensions can add human-only difficulty annotations and structure-aware formula equivalence metrics. 6 Conclusion We introduced OmniHandwritingOCR, a diagnostic benchmark and unified evaluation protocol for handwritten OCR in MLLM/OCR systems. By integrating multilingual handwritten text with single- line and difficulty-stratified multi-line formulas, the benchmark tests capabilities that are underrepresented in existing OCR eval- uations. Our evaluation of thirteen systems shows that aggregate performance remains limited, model rankings vary across language and formula settings, and structural complexity causes substantial degradation even for the strongest models. The fact-based annota- tion policy further exposes hallucinated correction, a failure mode that is especially important for generative OCR systems. These findings suggest that robust handwritten OCR for knowledge ex- traction requires evaluation protocols that are task-conditioned, structure-aware, and grounded in faithful transcription rather than plausible correction. GenAI Usage Disclosure Generative AI tools were used in this research in two roles. First, vision-language models were used for pre-annotation as described in Section 3. Second, GenAI-based coding assistants were used to assist with code development for conducting the experiments. OmniHandwritingOCR: A Diagnostic Benchmark for Evaluating Multimodal LLMs in Handwritten OCR ScenariosCIKM ’26, November 7-11, 2026, Rome, ITALY References [1]Robert H. Anderson. 1967. Syntax-directed recognition of hand-printed two- dimensional mathematics. In Symposium on Interactive Systems for Experimental Applied Mathematics: Proceedings of the Association for Computing Machinery Inc. Symposium (Washington, D.C.). Association for Computing Machinery, New York, NY, USA, 436–459. doi:10.1145/2402536.2402585 [2] Ahmad-Montaser Awal, Harold Mouchre, and Christian Viard-Gaudin. 2014. A global learning approach for an online handwritten mathematical expression recognition system. Pattern Recogn. Lett. 35, C (Jan. 2014), 68–77. [3]Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2016. Neural Machine Translation by Jointly Learning to Align and Translate. doi:10.48550/arXiv.1409. 0473 arXiv:1409.0473 [cs]. [4]Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond. doi:10.48550/ arXiv.2308.12966 arXiv:2308.12966 [cs]. [5] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. 2025. Qwen2.5-VL Technical Report. arXiv:2502.13923 [cs.CV] https://arxiv.org/abs/2502.13923 [6]Leonard E Baum and Ted Petrie. 1966. Statistical inference for probabilistic functions of finite state Markov chains. The annals of mathematical statistics 37, 6 (1966), 1554–1563. [7]Nancy Chinchor. 1992. MUC-4 evaluation metrics. In Proceedings of the 4th Conference on Message Understanding (McLean, Virginia) (MUC4 ’92). Association for Computational Linguistics, USA, 22–29. doi:10.3115/1072064.1072067 [8]Cheng Cui, Ting Sun, Suyin Liang, Tingquan Gao, Zelun Zhang, Jiaxuan Liu, Xueqing Wang, Changda Zhou, Hongen Liu, Manhui Lin, Yue Zhang, Yubo Zhang, Handong Zheng, Jing Zhang, Jun Zhang, Yi Liu, Dianhai Yu, and Yan- jun Ma. 2025. PaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Model. doi:10.48550/arXiv.2510.14528 arXiv:2510.14528 [cs]. [9]Ling Fu, Zhebin Kuang, Jiajun Song, Mingxin Huang, Biao Yang, Yuzhe Li, Linghao Zhu, Qidi Luo, Xinyu Wang, Hao Lu, Zhang Li, Guozhi Tang, Bin Shan, Chunhui Lin, Qi Liu, Binghong Wu, Hao Feng, Hao Liu, Can Huang, Jingqun Tang, Wei Chen, Lianwen Jin, Yuliang Liu, and Xiang Bai. 2025. OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning. doi:10.48550/arXiv.2501.00321 arXiv:2501.00321 [cs]. [10] Philippe Gervais, Anastasiia Fadeeva, and Andrii Maksai. 2025. MathWriting: A Dataset For Handwritten Mathematical Expression Recognition. doi:10.48550/ arXiv.2404.10690 arXiv:2404.10690 [cs]. [11]Alex Graves and Jürgen Schmidhuber. 2008. Offline handwriting recognition with multidimensional recurrent neural networks. In Proceedings of the 22nd International Conference on Neural Information Processing Systems (Vancouver, British Columbia, Canada) (NIPS’08). Curran Associates Inc., Red Hook, NY, USA, 545–552. [12]Emmanuèle Grosicki, Matthieu Carré, Jean-Marie Brodin, and Edouard Geoffrois. 2009. Results of the RIMES Evaluation Campaign for Handwritten Mail Process- ing. In 2009 10th International Conference on Document Analysis and Recognition. 941–945. doi:10.1109/ICDAR.2009.224 ISSN: 2379-2140. [13]Tobias Grüning, Roger Labahn, Markus Diem, Florian Kleber, and Stefan Fiel. 2017. READ-BAD: A New Dataset and Evaluation Scheme for Baseline Detection in Archival Documents. arXiv:1705.03311 [cs.CV] https://arxiv.org/abs/1705.03311 [14]Tobias Grüning, Roger Labahn, Markus Diem, Florian Kleber, and Stefan Fiel. 2017. READ-BAD: A New Dataset and Evaluation Scheme for Baseline Detection in Archival Documents. arXiv:1705.03311 [cs.CV] https://arxiv.org/abs/1705.03311 [15]Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation 9, 8 (1997), 1735–1780. doi:10.1162/neco.1997.9.8.1735 [16]M. J. Hunt. 1990. The Word Error Rate: A Measure of Performance for Automatic Speech Recognition. In Proceedings of the DARPA Workshop on Speech and Natural Language. Morgan Kaufmann, Hidden Valley, Pennsylvania, USA, 20–25. https: //aclanthology.org/H90-1001/ [17]RP Ram Kumar, A Chandra Prasad, K Vishnuvardhan, K Bhuvanesh, and San- jeev Dhama. 2023. Automated Handwritten Text Recognition. In E3S Web of Conferences, Vol. 430. EDP Sciences, 01022. [18]VI Lcvenshtcin. 1966. Binary coors capable or ‘correcting deletions, insertions, and reversals. In Soviet physics-doklady, Vol. 10. [19]Alex W. C. Lee, Jonathan Chung, and Marco Lee. 2021. GNHK: A Dataset for English Handwriting in the Wild. In Document Analysis and Recognition – ICDAR 2021: 16th International Conference, Lausanne, Switzerland, September 5–10, 2021, Proceedings, Part IV (Lausanne, Switzerland). Springer-Verlag, Berlin, Heidelberg, 399–412. doi:10.1007/978-3-030-86337-1_27 [20] Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, and Furu Wei. 2022. TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models. doi:10.48550/arXiv.2109.10282 arXiv:2109.10282 [cs]. [21]Zhang Li, Yuliang Liu, Qiang Liu, Zhiyin Ma, Ziyang Zhang, Shuo Zhang, Zidun Guo, Jiarui Zhang, Xinyu Wang, and Xiang Bai. 2025. MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigm. doi:10.48550/ arXiv.2506.05218 arXiv:2506.05218 [cs]. [22]Cheng-Lin Liu, Fei Yin, Da-Han Wang, and Qiu-Feng Wang. 2011. CASIA Online and Offline Chinese Handwriting Databases. In 2011 International Conference on Document Analysis and Recognition. IEEE, Beijing, China, 37–41. doi:10.1109/ ICDAR.2011.17 [23]Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual Instruc- tion Tuning. arXiv:2304.08485 [cs.CV] https://arxiv.org/abs/2304.08485 [24]Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. 2024. DeepSeek-VL: Towards Real-World Vision-Language Understanding. arXiv:2403.05525 [cs.AI] https://arxiv.org/abs/ 2403.05525 [25]U.-V. Marti and H. Bunke. 2002. The IAM-database: an English sentence database for offline handwriting recognition. International Journal on Document Analysis and Recognition 5, 1 (Nov. 2002), 39–46. doi:10.1007/s100320200071 [26]Nanonets Research Team. 2025. Nanonets-OCR2: A model for transforming docu- ments into structured markdown with intelligent content recognition and semantic tagging. Technical Report. Nanonets. https://nanonets.com/research/nanonets- ocr-2/ [27]OpenAI. 2023. GPT-4V(ision) System Card. Technical Report. OpenAI. https: //cdn.openai.com/papers/GPTV_System_Card.pdf [28] OpenAI. 2024. GPT-4o System Card. Technical Report. OpenAI. https://cdn. openai.com/gpt-4o-system-card.pdf [29] Linke Ouyang, Yuan Qu, Hongbin Zhou, Jiawei Zhu, Rui Zhang, Qunshu Lin, Bin Wang, Zhiyuan Zhao, Man Jiang, Xiaomeng Zhao, Jin Shi, Fan Wu, Pei Chu, Minghao Liu, Zhenxiang Li, Chao Xu, Bo Zhang, Botian Shi, Zhongying Tu, and Conghui He. 2025. OmniDocBench: Benchmarking Diverse PDF Docu- ment Parsing with Comprehensive Annotations. doi:10.48550/arXiv.2412.07626 arXiv:2412.07626 [cs]. [30]Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2001. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL ’02. Association for Computational Linguistics, Philadelphia, Pennsylvania, 311. doi:10.3115/1073083.1073135 [31] QwenTeam. 2025.Qwen3-VL: Sharper Vision, Deeper Thought, BroaderAction.Webpage.https://qwen.ai/blog?id= 99f0335c4ad9f6153e517418d48535ab6d8afef&from=research.latest- advancements-list [32]Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2009. The Graph Neural Network Model. IEEE Transactions on Neural Networks 20, 1 (2009), 61–80. doi:10.1109/TNN.2008.2005605 [33] Baoguang Shi, Xiang Bai, and Cong Yao. 2015. An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition. doi:10.48550/arXiv.1507.05717 arXiv:1507.05717 [cs]. [34] TAL Education Group. 2023. ICDAR 2023 Competition on Recognition of Multi- line Handwritten Mathematical Expressions. Web page. https://ai.100tal.com/ icdar Accessed on 2025-11-12. [35]Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieil- lard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean- bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Robert Busa-Fekete, Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, Iain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan-Thorsten Peter, Danila Sinopalnikov, Surya Bhupatiraju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky, Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma, Ab- heesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alex Feng, Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic, Amit Vadi, András György, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang, Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Choquette-Choo, C. J. Carey, Cormac Brick, Daniel Deutsch, Danielle Eisenbud, Dee Cattle, Derek Cheng, Dimitris Paparas, Divyashree Shivakumar Sreepathi- halli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna Klimczak-Plucińska, Harman Singh, Harsh Mehta, Harshal Tushar Lehri, Hussein Hazimeh, Ian Ballantyne, Idan Szpektor, Ivan Nardini, Jean Pouget- Abadie, Jetha Chan, Joe Stanton, John Wieting, Jonathan Lai, Jordi Orbay, Joseph Fernandez, Josh Newlan, Ju-yeong Ji, Jyotinder Singh, Kat Black, Kathy Yu, Kevin Hui, Kiran Vodrahalli, Klaus Greff, Linhai Qiu, Marcella Valentine, Marina Coelho, Marvin Ritter, Matt Hoffman, Matthew Watson, Mayank Chaturvedi, Michael Moynihan, Min Ma, Nabila Babar, Natasha Noy, Nathan Byrd, Nick Roy, Nikola Momchev, Nilay Chauhan, Noveen Sachdeva, Oskar Bunyan, Pankil Botarda, CIKM ’26, November 7-11, 2026, Rome, ITALYZinuo Guo, Min Zhang, and Bo Jiang Paul Caron, Paul Kishan Rubenstein, Phil Culliton, Philipp Schmid, Pier Giuseppe Sessa, Pingmei Xu, Piotr Stanczyk, Pouya Tafti, Rakesh Shivanna, Renjie Wu, Renke Pan, Reza Rokni, Rob Willoughby, Rohith Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Girgin, Shariq Iqbal, Shashir Reddy, Shruti Sheth, Siim Põder, Sijal Bhatnagar, Sindhu Raghuram Panyam, Sivan Eiger, Susan Zhang, Tianqi Liu, Trevor Yacovone, Tyler Liechty, Uday Kalra, Utku Evci, Vedant Misra, Vincent Roseberry, Vlad Feinberg, Vlad Kolesnikov, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlam Chow, Yuvein Zhu, Zichuan Wei, Zoltan Egyed, Victor Cotruta, Minh Giang, Phoebe Kirk, Anand Rao, Kat Black, Nabila Babar, Jes- sica Lo, Erica Moreira, Luiz Gustavo Martins, Omar Sanseviero, Lucas Gonzalez, Zach Gleicher, Tris Warkentin, Vahab Mirrokni, Evan Senter, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov, Noah Fiedel, Noam Shazeer, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Jean-Baptiste Alayrac, Rohan Anil, Dmitry, Lepikhin, Sebastian Borgeaud, Olivier Bachem, Armand Joulin, Alek Andreev, Cassidy Hardin, Robert Dadashi, and Léonard Hussenot. 2025. Gemma 3 Technical Report. doi:10.48550/arXiv.2503.19786 arXiv:2503.19786 [cs]. [36] Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, Congcong Wang, Dehao Zhang, Dikang Du, Dongliang Wang, Enming Yuan, Enzhe Lu, Fang Li, Flood Sung, Guangda Wei, Guokun Lai, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang, Haoning Wu, Haotian Yao, Haoyu Lu, Heng Wang, Hongcheng Gao, Huabin Zheng, Jiaming Li, Jianlin Su, Jianzhou Wang, Jiaqi Deng, Jiezhong Qiu, Jin Xie, Jinhong Wang, Jingyuan Liu, Junjie Yan, Kun Ouyang, Liang Chen, Lin Sui, Longhui Yu, Mengfan Dong, Mengnan Dong, Nuo Xu, Pengyu Cheng, Qizheng Gu, Runjie Zhou, Shaowei Liu, Sihan Cao, Tao Yu, Tianhui Song, Tongtong Bai, Wei Song, Weiran He, Weixiao Huang, Weixin Xu, Xiaokun Yuan, Xingcheng Yao, Xingzhe Wu, Xinxing Zu, Xinyu Zhou, Xinyuan Wang, Y. Charles, Yan Zhong, Yang Li, Yangyang Hu, Yanru Chen, Yejie Wang, Yibo Liu, Yibo Miao, Yidao Qin, Yimin Chen, Yiping Bao, Yiqin Wang, Yongsheng Kang, Yuanxin Liu, Yulun Du, Yuxin Wu, Yuzhi Wang, Yuzi Yan, Zaida Zhou, Zhaowei Li, Zhejun Jiang, Zheng Zhang, Zhilin Yang, Zhiqi Huang, Zihao Huang, Zijia Zhao, and Ziwei Chen. 2025. Kimi-VL Technical Report. doi:10.48550/arXiv.2504.07491 arXiv:2504.07491 [cs] version: 1. [37] Bin Wang, Zhuangcheng Gu, Guang Liang, Chao Xu, Bo Zhang, Botian Shi, and Conghui He. 2024. UniMERNet: A Universal Network for Real-World Mathemati- cal Expression Recognition. arXiv:2404.15254 [cs.CV] https://arxiv.org/abs/2404. 15254 [38] Bin Wang, Chao Xu, Xiaomeng Zhao, Linke Ouyang, Fan Wu, Zhiyuan Zhao, Rui Xu, Kaiwen Liu, Yuan Qu, Fukai Shang, Bo Zhang, Liqun Wei, Zhihao Sui, Wei Li, Botian Shi, Yu Qiao, Dahua Lin, and Conghui He. 2024. MinerU: An Open-Source Solution for Precise Document Content Extraction. doi:10.48550/arXiv.2409.18839 arXiv:2409.18839 [cs]. [39] Haoran Wei, Chenglong Liu, Jinyue Chen, Jia Wang, Lingyu Kong, Yanming Xu, Zheng Ge, Liang Zhao, Jianjian Sun, Yuang Peng, Chunrui Han, and Xiangyu Zhang. 2024. General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model. doi:10.48550/arXiv.2409.01704 arXiv:2409.01704 [cs]. [40]Haoran Wei, Yaofeng Sun, and Yukun Li. 2025. DeepSeek-OCR: Contexts Optical Compression. doi:10.48550/arXiv.2510.18234 arXiv:2510.18234 [cs]. [41]Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang, Liang Zhao, Yisong Wang, and Chong Ruan. 2024. DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding. doi:10.48550/arXiv.2412.10302 arXiv:2412.10302 [cs]. [42]Yejing Xie, Harold Mouchère, Foteini Simistira Liwicki, Sumit Rakesh, Rajkumar Saini, Masaki Nakagawa, Cuong Tuan Nguyen, and Thanh-Nghia Truong. 2023. ICDAR 2023 CROHME: Competition on Recognition of Handwritten Mathemati- cal Expressions. In Document Analysis and Recognition - ICDAR 2023, Gernot A. Fink, Rajiv Jain, Koichi Kise, and Richard Zanibbi (Eds.). Vol. 14188. Springer Nature Switzerland, Cham, 553–565. doi:10.1007/978-3-031-41679-8_33 Series Title: Lecture Notes in Computer Science. [43]Zhibo Yang, Jun Tang, Zhaohai Li, Pengfei Wang, Jianqiang Wan, Humen Zhong, Xuejing Liu, Mingkun Yang, Peng Wang, Shuai Bai, LianWen Jin, and Junyang Lin. 2024. C-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacy. doi:10.48550/arXiv.2412.02210 arXiv:2412.02210 [cs]. [44] Ye Yuan, Xiao Liu, Wondimu Dikubab, Hui Liu, Zhilong Ji, Zhongqin Wu, and Xi- ang Bai. 2022. Syntax-Aware Network for Handwritten Mathematical Expression Recognition. doi:10.48550/arXiv.2203.01601 arXiv:2203.01601 [cs]. [45]Hesuo Zhang, Lingyu Liang, and Lianwen Jin. 2020. SCUT-HCCDoc: A new benchmark dataset of handwritten Chinese text in unconstrained camera- captured documents. Pattern Recognition 108, Article 107559 (Dec. 2020), 107559 pages. doi:10.1016/j.patcog.2020.107559 [46]Jianshu Zhang, Jun Du, and Lirong Dai. 2017. A GRU-based Encoder-Decoder Approach with Attention for Online Handwritten Mathematical Expression Recognition. doi:10.48550/arXiv.1712.03991 arXiv:1712.03991 [cs]. [47]Jianshu Zhang, Jun Du, Yongxin Yang, Yi-Zhe Song, Si Wei, and Lirong Dai. 2020. A tree-structured decoder for image-to-markup generation. In Proceedings of the 37th International Conference on Machine Learning (ICML’20, Vol. 119). JMLR.org, 11076–11085. [48]Wenqi Zhao, Liangcai Gao, Zuoyu Yan, Shuai Peng, Lin Du, and Ziyin Zhang. 2021. Handwritten Mathematical Expression Recognition with Bidirectionally Trained Transformer. arXiv:2105.02412 [cs.CV] https://arxiv.org/abs/2105.02412 [49]Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Nianchen Deng, Songze Li, Yinan He, Tan Jiang, Jiapeng Luo, Yi Wang, Conghui He, Botian Shi, Xingcheng Zhang, Wenqi Shao, Junjun He, Yingtong Xiong, Wenwen Qu, Peng Sun, Penglong Jiao, Han Lv, Lijun Wu, Kaipeng Zhang, Huipeng Deng, Jiaye Ge, Kai Chen, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. 2025. InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models. doi:10.48550/arXiv.2504.10479 arXiv:2504.10479 [cs]. [50]Jianhua Zhu, Wenqi Zhao, Yu Li, Xingjian Hu, and Liangcai Gao. 2024. TAMER: Tree-Aware Transformer for Handwritten Mathematical Expression Recognition. doi:10.48550/arXiv.2408.08578 arXiv:2408.08578 [cs].