Paper deep dive
Full-Page Optical Music Recognition of Handwritten Monophonic Scores
Adrian Rosello, Antonio Ríos-Vila, David Rizo, Jorge Calvo-Zaragoza
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 9/9/2026, 4:40:51 AM
Summary
This paper investigates full-page Optical Music Recognition (OMR) for handwritten monophonic scores, addressing the limitations of traditional staff-segmentation pipelines. The authors evaluate Transformer-based architectures, specifically the Sheet Music Transformer (SMT) and the Unfolding method, against a CRNN-based baseline. A key contribution is a synthetic data generator capable of producing both typeset and handwritten styles to study pretraining impacts. Experiments on three real datasets (Jazzmus, FMT-C, FMT-M) demonstrate that SMT significantly outperforms baselines. Crucially, the study finds that pretraining on typeset synthetic data is as effective as, or better than, handwritten synthetic data, suggesting that learning structural layout conventions is more important than visual similarity to handwriting.
Entities (11)
Relation Signals (9)
Sheet Music Transformer → evaluatedon → FMT-C
confidence 95% · Experiments on three real handwritten datasets provide a comparative evaluation... FMT-C
Sheet Music Transformer → evaluatedon → FMT-M
confidence 95% · Experiments on three real handwritten datasets provide a comparative evaluation... FMT-M
Sheet Music Transformer → evaluatedon → Jazzmus
confidence 95% · Experiments on three real handwritten datasets provide a comparative evaluation... Jazzmus corpus
Sheet Music Transformer → outperforms → Unfolding method
confidence 95% · The experimental outcomes presented in Table 2, indicate that the SMT architecture consistently outperforms other baselines across all tested corpora
Sheet Music Transformer → outperforms → CRNN
confidence 95% · The experimental outcomes presented in Table 2, indicate that the SMT architecture consistently outperforms other baselines across all tested corpora
Typeset Pretraining → matchesorsurpasses → Handwritten Pretraining
confidence 93% · A key finding is that typeset pretraining matched or surpassed handwritten pretraining on all datasets
Smashcima → usedfor → Handwritten Style Generation
confidence 92% · For obtaining a handwritten style, we rely on the Smashcima engraver [13], a synthesizer specifically designed to generate full-page manuscript-style scores
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Full-page end-to-end Optical Music Recognition seeks to transcribe entire music pages directly into symbolic notation, avoiding the limitations of traditional pipelines that rely on accurate staff segmentation. Recent Transformer-based architectures have achieved strong performance on typeset scores, relying on large-scale synthetic data for pretraining. However, their applicability to handwritten music remains largely unexplored. In this work, we study full-page transcription on handwritten monophonic collections and analyze the impact of synthetic pretraining in this setting. To investigate which factors are most relevant during pretraining, we introduce a generator capable of producing visually coherent full-page scores in both typeset and handwritten styles. Experiments on three real handwritten datasets provide a comparative evaluation of several full-page pipelines and different synthetic pretraining strategies. The results suggest that the benefits of synthetic pretraining are primarily associated with learning structural layout conventions rather than with visual similarity to the target handwriting.
Tags
Links
- Source: https://arxiv.org/abs/2609.05662v1
- Canonical: https://arxiv.org/abs/2609.05662v1
Trouble viewing inline? Open PDF directly →
Full Text
24,136 characters extracted from source content.
Expand or collapse full text
Full-Page Optical Music Recognition of Handwritten Monophonic Scores Journal: Pattern Recognition Letters Adrian Rosello Email: email address Affiliation: Pattern Recognition and Artificial Intelligence group, University of Alicante, Alicante, Spain Antonio Ríos-Vila Affiliation: Pattern Recognition and Artificial Intelligence group, University of Alicante, Alicante, Spain David Rizo Affiliation: Pattern Recognition and Artificial Intelligence group, University of Alicante, Alicante, Spain Affiliation: Instituto Superior de Enseñanzas Artísticas de la Comunidad Valenciana, Alicante, Spain Jorge Calvo-Zaragoza Affiliation: Pattern Recognition and Artificial Intelligence group, University of Alicante, Alicante, Spain Abstract Full-page end-to-end Optical Music Recognition seeks to transcribe entire music pages directly into symbolic notation, avoiding the limitations of traditional pipelines that rely on accurate staff segmentation. Recent Transformer-based architectures have achieved strong performance on typeset scores, relying on large-scale synthetic data for pretraining. However, their applicability to handwritten music remains largely unexplored. In this work, we study full-page transcription on handwritten monophonic collections and analyze the the impact of synthetic pretraining in this setting. To investigate which factors are most relevant during pretraining, we introduce a generator capable of producing visually coherent full-page scores in both typeset and handwritten styles. Experiments on three real handwritten datasets provide a comparative evaluation of several full-page pipelines and different synthetic pretraining strategies. The results suggest that the benefits of synthetic pretraining are primarily associated with learning structural layout conventions rather than with visual similarity to the target handwriting. Keywords: Optical Music Recognition , Full-Page Transcription , Synthetic Music Generation , Monophonic Scores 1 Introduction Handcrafted sheet music transcription is a tedious process that demands specialized knowledge of the music notation. Yet without the representation obtained from this process, a music document cannot be searched, edited, or analyzed, regardless of whether a digital image of it exists. With recent developments in deep learning, Optical Music Recognition (OMR) has emerged as an alternative to this manual effort, offering a scalable path to digitize the vast collections of music documents that remain inaccessible [3]. The end goal of OMR is the recognition of full music pages, which requires models to handle both layout understanding and symbol recognition across the entire page. For a long time, this complexity made holistic full-page transcription a challenging task. Music sheets are structured documents composed of multiple staves, each containing a sequence of musical symbols that must be read in order. Recognizing the full page means not only identifying those symbols correctly but also understanding how the staves are arranged and how they relate to each other. Given this difficulty, early approaches converged toward a simpler proxy: a two-step process where layout analysis first locates all staves, and each one is then transcribed individually. This setup, however, creates a snowball effect where layout analysis errors propagate through the system. Furthermore, treating each staff in isolation prevents the model from leveraging inter-staff context, such as consistent key and time signatures. Despite these limitations, staff-level approaches have shown strong recognition performance, with most systems adopting the Convolutional Recurrent Neural Network (CRNN) architecture trained with Connectionist Temporal Classification (CTC) loss [4, 17, 11], combining convolutional layers for feature extraction with recurrent layers for temporal modeling. Recent works have taken a step further by exploring holistic full-page approaches that transcribe entire sheets in a single step, removing this dependency altogether. Early efforts in this direction led to the so-called Unfolding method [10], which trains models to flatten multi-staff pages into sequential representations. However, this flattening causes a loss of spatial context, limiting the method to specific domains. To address this, subsequent works adopted Transformer-based architectures capable of transcribing without spatial constraints. A noteworthy contribution was the work of Ríos-Vila et al. [15], who proposed the Sheet Music Transformer (SMT), an encoder-decoder architecture trained with curriculum learning on large-scale synthetic data, achieving state-of-the-art full-page transcription even under limited data conditions. Yet, these advances have been largely confined to very specific scenarios. The Unfolding method has been primarily evaluated on mensural notation, a historically simpler form of music that does not reflect the notational complexity of modern scores. The SMT, on the other hand, has been designed and optimized for polyphonic typeset modern music, leaving other sources and styles largely unexplored. This raises a question: how well do these methods generalize beyond the narrow settings in which they were developed? Indeed, this concern is backed by our preliminary experiments. When applying these methods directly to handwritten monophonic sources, both the Unfolding and SMT models fail to transfer successfully, even after fine-tuning on the target domain (Fig. 1). This gap demonstrates that full-page models cannot be directly applied to new scenarios without careful consideration of the target domain. Figure 1: 10-fold error rates after directly adjusting the Unfolding [10] and SMT [15] models to the target handwritten corpora. Although the effect of handwritten data on full-page models remains largely unknown, thanks to recent advances in synthetic music generation, it is now possible to study musical styles and textures that have never been explored in the context of full-page recognition. For instance, the Smashcima [13] generator, which has not yet been studied in an experimental setting, now enables the generation of realistic handwritten music scores, opening new frontiers to study this phenomenon at a larger scale. In this work, we present the first study of full-page OMR on handwritten monophonic collections. We investigate whether the difficulty of this transfer stems from the visual variability of handwritten notation or from the structural properties of monophonic music. To this end, we introduce a tailored full-page synthetic generator capable of producing scores in both typeset and handwritten styles, enabling controlled pretraining experiments that reveal what truly matters when adapting full-page models to this unexplored scenario.† The code of this paper will be released upon acceptance of this paper. This footnote will be changed to reflect the link to the online code repository in GitHub. 2 Methodology The goal of full-page transcription is to, given a set of full-page images X, retrieve the sequence of tokens =(t1,t2,…,tn)t= (t_1,t_2,…,t_n ) of the music embedded in a single image x∈x . Each possible value of any token tit_i is represented in a finite vocabulary Σ , being Σ∗ ^* all possible combinations of sequences from this vocabulary. We assume that there exists a function f:→Σ∗f:X→ ^* that relates each sheet image x∈x with its sequence representation t∈Σ∗t∈ ^*. Evaluated techniques will try to approximate this function f as f f by selecting the most probable token at each position in the sequence given any input image x∈x . The way this likelihood is computed depends on the decoding strategy: CTC greedy decoding. In CTC-based models, the output of the network is a sequence of frame-wise distributions over an extended vocabulary Σ′=Σ∪∅ \!=\! \,∪\, , which includes a blank symbol. Given an input image x, the model produces a sequence of predictions π=(π1,π2…,πk)π~=~( _1, _2…, _k) of length K, where each πk∈Σ′ _k∈ is chosen independently of the vocabulary Σ′ based on its predicted likelihood at frame k. The final transcription t is obtained by applying the CTC collapsing function ℬ(⋅)B (· ): f^(x)=ℬ(π),withπk=argmaxπ∈Σ∗P(π∣x,k) f(x)=B(π),with~ _k= _π∈ ^*P(π x,k) Here, k indexes the frame-wise outputs of the network, while the resulting sequence t of length n≤Kn≤ K contains the actual predicted tokens after merging repeated symbols and removing blanks. Autoregressive decoding. In autoregressive models, the sequence is generated token by token, each conditioned on the input image and all previously predicted tokens. The model calculates the output probability as a product of conditional probabilities: f^(x)=(t^1,…,t^n),witht^i=argmaxt∈Σ∗P(t∣x,t^<i) f(x)=( t_1,…, t_n),with~ t_i= _t∈ ^*P(t x, t_<\,i) where t^<i=(t^1,…,t^i−1) t_<\,i=( t_1,…, t_i-1) is the predicted prefix up to position i−1i-1. During training, the model is optimized using a teacher forcing policy, where the true prefix t<it_<\,i is used instead. 3 Experimental Setup This section describes the experimental setup, including the evaluated pipelines, the synthetic data generation process, the datasets, and the evaluation protocol. 3.1 Two-steps Pipeline To compare with staff-level architectures, we adopt a segmentation-based strategy, which we will refer to from now on as CRNNfCRNN^f. This approach applies first a layout analysis with YOLOv11 [6] to segment the input page into individual staves, which are then concatenated into a single sequence for the CRNN model. However, this concatenation produces significantly longer sequences than those in standard staff-level training. Since the CRNN uses a bidirectional LSTM with limited long-range dependency modeling, this setup is suboptimal. Thus, CRNNfCRNN^f provides a fair but constrained adaptation of CRNN to full-page recognition. Figure 2: Diagram of the two-step pipeline used. 3.2 Holistic Pipelines We evaluate two segmentation-free full-page approaches: the Unfolding method [10], trained directly on the target corpora, and the SMT architecture [15], which follows a curriculum-based pretraining strategy on synthetic data of incremental difficulty before fine-tuning on the real datasets. Figure 3: Diagrams of the holistic pipelines used. 3.3 Label Representation The choice of label representation is a trade-off between vocabulary complexity and sequence length. While agnostic encodings focus on graphical primitives, semantic representations like MusicXML or MEI [7, 9] capture musical meaning but are often too verbose for sequence-to-sequence modeling. In this work, we adopt the basic extended **kern (**bekern) representation [15], which decomposes **kern tokens into atomic components separated by delimiters. Conversion from **bekern to **kern is fully reversible through naive preprocessing tools [5], ensuring compatibility with existing resources. 3.4 Synthetic Data Generation To pretrain the SMT architecture, we introduce a full-page generator tailored for synthetic monophonic sheets, built on the RISM incipits that serve as the source for the PrIMuS corpus [4]. Combining RISM staves naively into full pages can produce notational inconsistencies across staves—such as mismatched clefs, key signatures, or time signatures—leading to visual artifacts in the rendered output. To address this, the generator segments each incipit by measure and indexes them according to three musical attributes: key signature, time signature, and clef. These attributes are enforced to remain constant across all staves on a page, preventing artifacts such as notes rendered outside the staff range. Once a set of measures has been selected, a synthetic page is generated in either typeset or handwritten style (Fig.4). The goal is to produce musically coherent scores that are consistent from a visual and syntactic conventions point of view, not to generate musically meaningful compositions in any specific genre or style. (a) Typeset style (b) Handwritten style Figure 4: Representative examples of the generated music in both typeset (a) and handwritten (b) style. 3.4.1 Typeset Style To simulate a typeset style, we use Verovio [14] engraver. The toolkit provided by the engraver enables various modifications to the layout to generate different compositions. We can leverage this to provide randomization of such layouts to create more realistic distributions. To this end, applied variations modify page margins, content scale, and inter-system spacing. The engraver also allows creating textual elements like the work title or the author name, which we can also randomize using available text generators. By default, the pages yielded by this engraver are retrieved in plain black and white. To increase the resemblance to real manuscripts, a background texture is selected from a subset of images and then added to the scores. Even with the background textures, images are still far from realistic. For this reason, a set of transformations is applied using Augraphy [8]. Augraphy is a library that simulates paper-handling distortions by generating effects like ink bleeding, letterpress, ink shift, and background texturization. After this process, the generated pages achieve a visual quality much closer to scanned historical prints. 3.4.2 Handwritten Style For obtaining a handwritten style, we rely on the Smashcima engraver [13], a synthesizer specifically designed to generate full-page manuscript-style scores from symbolic encodings. Unlike traditional engravers, Smashcima produces glyphs that imitate the irregularities and spatial variability characteristic of handwritten notation. Given a MusicXML input, it renders a complete page while preserving detailed semantic information, including glyph locations, staff associations, and notational structure. Since the original RISM data is encoded in **kern, we convert the synthesized output into MusicXML using the converter2111 1 https://github.com/gregchapman-dev/converter21 python library, a tool built on music2122 2 https://w.music21.org/music21docs/ that provides bidirectional converters between many semantic encodings. We use the converted MusicXML to generate the synthetic image and leverage the **kern sequence to obtain the label sequence. 3.5 Evaluation Corpora To assess the performance of the proposed strategies under realistic conditions, we evaluate them on three real full-page music datasets 33 3 Due to the resolution of full-page images, examples of each dataset are depicted in the supplementary material.. These collections differ in layout, writing style, and level of degradation, offering a representative benchmark. - Jazzmus corpus [12]: a collection of 293 handwritten jazz lead sheets gathered from students and professional musicians, characterized by irregular handwriting styles, strike-through corrections, diverse capture conditions, and a wide variety of chord symbol conventions. - FMT-C and FMT-M corpora [16]: two partitions derived from a collection of four groups of handwritten score sheets of popular Spanish songs, notated in modern music notation. The source material originates from the Fondo de Música Tradicional (IMF-CSIC). FMT-C comprises vertically oriented images with a significant presence of textual elements, while FMT-M consists of landscape-format pages that exhibit signs of paper aging, such as stains and yellowing. Name FMT-C FMT-M Jazzmus Vocabulary size 60 75 96 Number of pages 86 205 163 Table 1: Statistics of the real full-page datasets. 3.6 Evaluation Protocol For evaluating the performance of the benchmarked models in full-page transcription, we resorted to the Symbol Error Rate (SER) figure of merit. This metric, widely used in previous works [2, 17, 1], measures the minimum number of operations—referring as operations to insertions, modifications, or eliminations of symbols—required to match the sequence predicted by the models to that of the ground truth. Due to the limited size of the real full-page datasets (Table 1), we adopt a 10-fold cross-validation strategy to ensure a fair comparison. Results are reported as the average SER across all folds, along with the standard deviation to capture variability. Figure 5: Comparison of the SMT, Unfolding, and CRNNfCRNN^fscenarios in terms of SER across 10 folds for the FMT-C, FMT-M, and Jazzmus datasets. 4 Results The experimental outcomes presented in Table 2, indicate that the SMT architecture consistently outperforms other baselines across all tested corpora regardless of the pretraining data style. FMT-C FMT-M Jazzmus CRNNfCRNN^f 45.7±27.845.7± 27.8 81.6±25.481.6± 25.4 63.1±2.863.1± 2.8 Unfolding 40.4±14.940.4± 14.9 68.3±18.068.3± 18.0 61.0±3.561.0± 3.5 SMTprSMT_pr 31.6±9.931.6 9.9 32.8±8.732.8± 8.7 26.9±3.426.9± 3.4 SMThwSMT_hw 38.4±10.438.4± 10.4 32.9±5.232.9 5.2 30.6±3.130.6 3.1 Table 2: 10-fold cross-validated SER (%) on FMT-C, FMT-M, and Jazzmus. Values report mean and standard deviation across folds. Best results per dataset are in bold. A more in-depth plot of the 10-fold evaluation is provided in Figure 5, where we can observe that, while the SMT models maintain a low average error, the CRNNfCRNN^f and Unfolding systems exhibit highly unstable behavior. In several folds, these baseline systems reach extreme error rates, whereas the SMT models generalize more reliably to varying layout conditions. However, the Unfolding approach remains competitive in specific monophonic contexts, as we can see by the results obtained in FMT-C, comparable to those obtained by SMThwSMT_hw. Regarding the influence of synthetic pretraining, the research reveals a disparity between visual fidelity and model accuracy. Although SMThwSMT_hw was developed to resemble closelly the handwritten domain, SMTprSMT_pr yielded a lower SER in both the FMT-C and Jazzmus collections, while achieving a marginal performance difference in the FMT-M dataset. These findings indicate that the primary contribution of synthetic pretraining lies in the acquisition of structural and semantic priors rather than the imitation of specific handwritten styles. 4.1 Qualitative Evaluation Figure 6 shows the output of SMTprSMT_pr on a sample from Jazzmus. The model correctly discards irrelevant content such as empty staves and the title heading, and recovers after local failures, resuming accurate transcription in subsequent measures. Most false negatives arise from non-renderable syntactic errors; false positives correspond to pitch misassignments. Locating these errors, however, is not straightforward. Mistakes are scattered across long symbolic sequences spanning multiple staves, making it difficult to assess where and why the model fails. Despite this being a central challenge in the digitization of music scores, no alternative to careful manual review currently exists. Figure 6: Qualitative result of the model SMTprSMT_pr performance on a test split of the 10k-fold of Jazzmus. Red denotes false positives, while blue represents false negatives. 5 Conclusions This research addressed the challenge of graphical variability in handwritten musical heritage. The SMT architecture consistently outperformed both the segmentation-based and unfolding baselines across three datasets with different layout conditions, as confirmed by 10-fold cross-validation. The other pipelines not only produced higher error rates but also showed large variation across folds. A key finding is that typeset pretraining matched or surpassed handwritten pretraining on all datasets, meaning that what matters most in synthetic data is page layout and notation conventions, not visual resemblance to handwriting. Future work should build on this premise when considering similar pretraining strategies. 6 Acknowledgements Work supported by grant CISEJI/2023/9 from “Programa para el apoyo a personas investigadoras con talento (Plan GenT) de la Generalitat Valenciana”. The first author is supported by grant CIACIF/2024/372 from “Programa I+D+i de la Generalitat Valenciana”. References [1] M. Alfaro-Contreras, A. Ríos-Vila, J. J. Valero-Mas, J. M. Iñesta, and J. Calvo-Zaragoza (2022) Decoupling music notation to improve end-to-end Optical Music Recognition. Pattern Recognition Letters. Cited by: §3.6. [2] A. Baró, C. Badal, and A. Fornés (2020) Handwritten Historical Music Recognition by Sequence-to-Sequence with Attention Mechanism. In International Conference on Frontiers in Handwriting Recognition, Cited by: §3.6. [3] J. Calvo-Zaragoza, J. H. Jr, and A. Pacha (2020) Understanding Optical Music Recognition. ACM Computing Surveys. Cited by: §1. [4] J. Calvo-Zaragoza and D. Rizo (2018) Camera-PrIMuS: Neural End-to-End Optical Music Recognition on Realistic Monophonic Scores. In International Society for Music Information Retrieval Conference, Cited by: §1, §3.4. [5] J. Cerveto-Serrano, D. Rizo, and J. Calvo-Zaragoza (2025) kernpy: a Humdrum **Kern Oriented Python Package for Optical Music Recognition Tasks. In Music Encoding Conference, Cited by: §3.3. [6] V. Dvořák, J. jr. Hajič, and J. Mayer (2024) Staff layout analysis using the yolo platform. In Proceedings of the 6th International Workshop on Reading Music Systems, Cited by: §3.1. [7] M. D. Good (2001) MusicXML: an internet-friendly format for sheet music. In Xml conference and expo, Cited by: §3.3. [8] A. Groleau, K. W. Chee, S. Larson, S. Maini, and J. Boarman (2023) Augraphy: a data augmentation library for document images. In International Conference on Document Analysis and Recognition, Cited by: §3.4.1. [9] A. Hankinson, P. Roland, and I. Fujinaga (2011) The music encoding initiative as a document-encoding framework. In International Society for Music Information Retrieval Conference, Cited by: §3.3. [10] J. C. Martinez-Sevilla, A. Rios-Vila, F. J. Castellanos, and J. Calvo-Zaragoza (2023) A holistic approach for aligned music and lyrics transcription. In International Conference on Document Analysis and Recognition, Cited by: Figure 1, Figure 1, §1, §3.2. [11] J. C. Martínez-Sevilla, A. Roselló, D. Rizo, and J. Calvo-Zaragoza (2023) On the Performance of Optical Music Recognition in the Absence of Specific Training Data. In International Society for Music Information Retrieval Conference, Cited by: §1. [12] J. C. Martinez-Sevilla, F. Foscarin, P. Garcia-Iasci, D. Rizo, J. Calvo-Zaragoza, and G. Widmer (2025) Optical music recognition of jazz lead sheets. Cited by: 1st item. [13] J. Mayer, P. Pecina, and Jr. Hajič (2025) Smashcima: full-page handwritten music document synthesizer. In International Conference on Digital Libraries for Musicology, Cited by: §1, §3.4.2. [14] L. Pugin, R. Zitellini, and P. Roland (2014) Verovio: A library for engraving MEI music notation into SVG. In International Society for Music Information Retrieval Conference, Cited by: §3.4.1. [15] A. Ríos-Vila, J. Calvo-Zaragoza, D. Rizo, and T. Paquet (2026) End-to-end full-page optical music recognition for pianoform sheet music. Int. J. Comput. Vision. Cited by: Figure 1, Figure 1, §1, §3.2, §3.3. [16] E. Ros-Fábregas (2021) Codified spanish music heritage through verovio: the online platforms fondo de música tradicional imf-csic and books of hispanic polyphony imf-csic.. Cited by: 2nd item. [17] M. Villarreal and J. A. Sánchez (2020) Handwritten Music Recognition Improvement through Language Model Re-interpretation for Mensural Notation. In International Conference on Frontiers in Handwriting Recognition, Cited by: §1, §3.6.