Paper deep dive
TUTTI: Toward generalizable audio-to-score transcription via fully synthesized data
Jianhuai Hu, Yashan Wang, Shangda Wu, Zhancheng Guo, Shijie Liang, Wuna Meng, Chuanqi Yang, Xiaobing Li, Feng Yu, Maosong Sun
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/2/2026, 5:29:51 AM
Summary
The paper introduces TUTTI, a Transformer-based Audio-to-Score (A2S) transcription model that leverages a large-scale, fully synthetic dataset called TuttiCorpus. By using a symbolic music generation model (NotaGen) to create over 363,000 multi-instrumentation audio-score pairs, the authors address the data scarcity bottleneck in A2S. TUTTI employs a standard encoder-decoder architecture with hierarchical decoding (patch-level and character-level) to transcribe audio spectrograms directly into ABC notation. The model demonstrates state-of-the-art performance and strong cross-instrument generalizability, outperforming existing baselines that rely on single-instrumentation data or hybrid CNN-RNN architectures.
Entities (9)
Relation Signals (8)
TUTTI → isbasedon → Transformer
confidence 96% · we employ a standard Transformer encoder-decoder architecture.
TUTTI → solves → Audio-to-Score
confidence 95% · TUTTI ... designed to perform generalizable multi-instrumentation A2S transcription.
TUTTI → uses → TuttiCorpus
confidence 95% · TUTTI ... pre-training paradigm driven by a purely synthetic, large-scale dataset ... TuttiCorpus dataset.
TUTTI → outputs → ABC Notation
confidence 94% · TUTTI directly decodes audio spectrogram features into ABC notation.
TuttiCorpus → generatedby → NotaGen
confidence 93% · we leverage NotaGen [25] ... to generate 363,610 unique ABC scores
TUTTI → outperforms → Zeng et al. [1]
confidence 90% · TUTTI outperforms previous approaches, establishing new overall state-of-the-art results
TUTTI → outperforms → Alfaro-Contreras et al. [2]
confidence 90% · TUTTI outperforms previous approaches, establishing new overall state-of-the-art results
TuttiCorpus → usestechnique → Expressive Performance Rendering
confidence 88% · we apply Expressive Performance Rendering (EPR) prior to audio synthesis ... to improve model generalization
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Generalizable Audio-to-Score (A2S) transcription is fundamentally constrained by the severe scarcity of high-quality, real-world paired data. Relying solely on existing human-annotated datasets often restricts the generalization of A2S models, limiting their efficacy primarily to single-instrumentation domains. To break this dependency on scarce real-world data, we introduce TUTTI (Transformer for Unified audio-To-score Transcription trained on Synthetic multi-Instrumentation Data), a pre-training paradigm driven by a purely synthetic, large-scale dataset. Rather than using human-composed scores, we leverage a symbolic music generation model to generate a massive, highly scalable multi-instrumentation corpus and create audio-score pairs with expressive acoustic characteristics. Capitalizing on the generated data, we employ a standard Transformer encoder-decoder architecture. We empirically demonstrate that pre-training a unified attention-based model on generated, multi-instrumentation data yields a consistently stronger foundational representation than single-instrumentation training. When fine-tuned with downstream real-world datasets, TUTTI outperforms previous approaches, establishing new overall state-of-the-art results across various A2S baselines. Notably, TUTTI shows remarkable cross-instrument transferability, effectively adapting to unseen instruments with highly competitive performance. The source code and the TuttiCorpus dataset will be made publicly available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2609.00640v1
- Canonical: https://arxiv.org/abs/2609.00640v1
Trouble viewing inline? Open PDF directly →
Full Text
38,560 characters extracted from source content.
Expand or collapse full text
TUTTI: TOWARD GENERALIZABLE AUDIO-TO-SCORE TRANSCRIPTION VIA FULLY SYNTHESIZED DATA Jianhuai Hu ∗†1 Yashan Wang ∗1 Shangda Wu ∗3 Zhancheng Guo 1 Shijie Liang 1 Wuna Meng 4 Chuanqi Yang 5 Xiaobing Li 1 Feng Yu 1 Maosong Sun †1,2 1 Central Conservatory of Music 2 Tsinghua University 3 Independent Researcher 4 Wuhan Conservatory of Music 5 Shanghai Conservatory of Music ∗ Equal contribution † Corresponding author hujianhuai@mail.ccom.edu.cn, sms@tsinghua.edu.cn ABSTRACT Generalizable Audio-to-Score (A2S) transcription is fun- damentally constrained by the severe scarcity of high- quality, real-world paired data. Relying solely on exist- ing human-annotated datasets often restricts the general- ization of A2S models, limiting their efficacy primarily to single-instrumentation domains. To break this depen- dency on scarce real-world data, we introduce TUTTI (Transformer for Unified audio-To-score Transcription trained on Synthetic multi-Instrumentation Data), a pre- training paradigm driven by a purely synthetic, large-scale dataset. Rather than using human-composed scores, we leverage a symbolic music generation model to generate a massive, highly scalable multi-instrumentation corpus and create audio-score pairs with expressive acoustic char- acteristics. Capitalizing on the generated data, we em- ploy a standard Transformer encoder-decoder architecture. We empirically demonstrate that pre-training a unified attention-based model on generated, multi-instrumentation data yields a consistently stronger foundational representa- tion than single-instrumentation training. When fine-tuned with downstream real-world datasets, TUTTI outperforms previous approaches, establishing new overall state-of-the- art results across various A2S baselines. Notably, TUTTI shows remarkable cross-instrument transferability, effec- tively adapting to unseen instruments with highly com- petitive performance. The source code and the TuttiCor- pus dataset will be made publicly available at https: //github.com/a-musiclover/TUTTI. 1. INTRODUCTION Audio-to-Score (A2S) transcription aims to extract score- level symbolic representations (e.g., ABC notation, © J. Hu, Y. Wang, S. Wu, Z. Guo, S. Liang, W. Meng, C. Yang, X. Li, F. Yu, and M. Sun. Licensed under a Creative Commons Attribution 4.0 International License (C BY 4.0). Attribution: J. Hu, Y. Wang, S. Wu, Z. Guo, S. Liang, W. Meng, C. Yang, X. Li, F. Yu, and M. Sun, “TUTTI: Toward generalizable audio-to-score transcription via fully synthesized data”, in Proc. of the 27th Int. Society for Music Information Retrieval Conf., Abu Dhabi, UAE, 2026. ** Kern) directly from acoustic audio signals.Cur- rently, A2S serves a dual purpose: it remains an indis- pensable tool for real-world music pedagogy and perfor- mance, while simultaneously acting as a pivotal data cu- ration pipeline for generative music models [1–4]. As the demand for large-scale audio-score paired datasets grows, the ability to automatically transcribe performances into formats such as ABC notation becomes a key enabler for training generative models. Although early approaches de- composed A2S into isolated subtasks such as harmony es- timation and note quantization, the paradigm has increas- ingly shifted towards end-to-end learning [5–11]. How- ever, despite recent architectural advances such as hierar- chical decoding [1] and customized Transformers [2], the development of a unified foundation model remains con- strained by several fundamental challenges. Data Scarcity Bottleneck Data availability is the most critical hurdle in current A2S research [1, 12]. The to- tal corpus of human-composed music is historically fi- nite. Compounding this limitation, high-quality real-world acoustic performances aligned perfectly with their sym- bolic scores are exceedingly rare and expensive to anno- tate [3, 13]. Consequently, relying on existing human- annotated datasets makes it nearly impossible to train an A2S model capable of tackling the vast complexity of di- verse, real-world musical compositions. Lack of Generalizability across Instrumentations Driven directly by this data scarcity, existing A2S mod- els are highly fragmented and task-specific. The field has largely focused on the “solo” scenario—models strictly tai- lored to single-instrumentation transcriptions such as solo piano, string quartets, saxophone, etc. [1–3,14]. To the best of our knowledge, a unified, generalizable A2S model ca- pable of robustly transcribing musical pieces across a wide and unconstrained variety of instrumentations has not yet been proposed. Task-Specific Architectural Complexity Given the historical lack of diverse, large-scale training data, the A2S field has naturally gravitated towards task-specific hybrid architectures. A common paradigm involves the use of convolutional neural networks (CNNs) as acous- tic feature extractors, coupled with recurrent neural net- arXiv:2609.00640v1 [cs.SD] 1 Sep 2026 works (RNNs) or attention mechanisms for sequence de- coding [1, 2, 11, 15]. These hybrid pipelines are highly ef- fective in data-scarce regimes because the strong local in- ductive biases of CNNs help prevent overfitting on small datasets [16, 17]. However, this reliance on specialized acoustic front-ends inherently increases architectural com- plexity. This raises an underexplored question in the A2S domain: With the introduction of massive, large-scale syn- thetic training data, is it possible to bypass customized acoustic front-ends and rely entirely on a unified, standard architecture? To answer this, we propose a paradigm shift from model-centric engineering to a data-centric approach. In this paper, we introduce TUTTI, a pure Transformer-based encoder-decoder framework designed to perform gener- alizable multi-instrumentation A2S transcription. Treat- ing A2S fundamentally as an “audio-to-text” sequence-to- sequence translation task [18], TUTTI directly decodes au- dio spectrogram features into ABC notation. Inspired by the success of pure attention-based models in Natural Lan- guage Processing [19–23] and their recent adaptations in symbolic music [24–26], we intentionally adopt a standard Transformer architecture.By incorporating techniques such as interleaved ABC notation and hierarchical patch- level and char-level decoders [24,25,27], pure Transformer architectures empowered by massive synthetic data can achieve highly competitive representations without strictly requiring customized CNN front-ends, highlighting that data scale is the primary driver of performance. Natu- rally, traditional CNN-based approaches stand to benefit similarly from such large-scale pre-training. To overcome the data bottleneck, we pre-train TUTTI on a massive syn- thetic dataset. Rather than relying on limited human com- positions, we leverage NotaGen [25] to procedurally create over 363,000 unique, multi-instrumentation scores in ABC notation, which are subsequently rendered into audio. By pre-training on this unconstrained corpus, TUTTI learns a powerful foundational representation of multi-instrument acoustics. The key contributions of our paper are as follows: • TUTTI, an attention-based generalizable encoder- decoder model that directly transcribes audio spec- trograms into ABC notation without relying on tra- ditional CNN feature extractors. • TuttiCorpus, a dataset of over 363,000 unique multi- instrumentation audio-score pairs. It enables the model to learn complex polyphonic structures across diverse musical textures through a purely generative data pipeline. • Our experimental results demonstrate that TUTTI achieves state-of-the-art performance across vari- ous instrumentations. Furthermore, we show that multi-instrumentation pre-training consistently out- performs single-domain training, and that the model exhibits generalizability to the unseen instrument (i.e., saxophone) which is entirely absent from the pre-training data. 2. RELATED WORK Early approaches to A2S transcription predominantly treated the problem as a pipeline of isolated subtasks, such as multi-pitch estimation followed by rhythm quantiza- tion [5–7]. While functional, these decomposition-based methods often suffered from error accumulation across dif- ferent stages. To mitigate this, the field has seen a notable paradigm shift towards end-to-end models [8]. The current standard for end-to-end A2S heavily re- lies on a hybrid convolutional recurrent neural network (CRNN) architecture within an encoder-decoder frame- work. Typically, a CNN front-end is deployed to extract local acoustic features from audio spectrograms, which are subsequently decoded into musical symbols by an RNN [1–3, 15]. Training strategies for these sequence labeling tasks generally diverge into two paths: utilizing Connectionist Temporal Classification (CTC) loss [9, 15] to bypass explicit input alignment, or adopting a Sequence- to-Sequence (Seq2Seq) framework [11] for direct target prediction. Despite improving overall transcription coher- ence, translating dense polyphonic music into structured score formats (e.g., ABC notation, ** Kern) remains a formidable challenge, with many models struggling to consistently maintain voicing structures across complex, unconstrained musical textures [15]. To address the hierarchical complexity of musical scores, recent research has explored more advanced de- coding mechanisms. Zeng et al. [1] introduced a Seq2Seq model with a hierarchical decoder capable of capturing both bar-level metadata and note-level sequences. To coun- teract the scarcity of real-world data, they pre-trained their model using an Expressive Performance Rendering (EPR) system to humanize the rendered data before fine-tuning on human piano recordings. Concurrently, recognizing the inherent limitations of RNNs in modeling long-term polyphonic dependencies, Alfaro-Contreras et al. [2] inte- grated a Transformer decoder into the A2S pipeline. They employed a specialized two-dimensional positional encod- ing to preserve frequency-time relationships from the CNN feature maps, achieving significant improvements in tran- scribing polyphonic string music. Despite these architectural innovations, existing end- to-end models remain fundamentally constrained by the scarcity of large-scale, diverse training data, heavily re- stricting their applicability to single-instrumentation sce- narios. Furthermore, the persistent reliance on CNN acous- tic front-ends introduces architectural complexity. 3. METHODOLOGY In this section, we detail the methodology behind TUTTI. Building upon recent advancements in symbolic music processing and generation [24, 25], we adapt the hierarchi- cal bar-stream patching strategy to the challenging cross- modal domain of A2S transcription. We first outline the structured ABC notation adopted for data representation (Section 3.1). We then detail the core architectural design of TUTTI, highlighting our continuous acoustic encoder Spectrogram ( 푇 × 퐹) Acoustic Adapter Audio Encoding Encoded Audio Features Linear Projection of Flattened Bar Patches [START]L:1/8M:6/8K:Ab(E2 F [EG].. ABCDecoding Sequence of Audio Embeddings ( 푇 × 푑 !"#$% ) Encoder- Decoder Cross Attention Transformer Musical EncoderAcoustic-level Transformer Encoder ( 푇 × 푑 !"#$% ) Patch-level Transformer Decoder Patch-level Transformer Decoder Patch-level Transformer Decoder Character-level Transformer Decoder Character-level Transformer Decoder Piano 8 6 8 6 Figure 1: Architectural overview of TUTTI. The model bridges the gap between continuous audio spectrograms and discrete symbolic scores through a Transformer encoder and a hierarchical decoder. and the cross-modal hierarchical decoder. 3.1 Data Representation We use ABC notation for symbolic music representation. Departing from the original plain ABC format, we adopt an advanced bar-stream patching strategy, along with an interleaved and reduced ABC notation format, to ensure a highly structured and sequence-efficient representation, as established in previous works [25–27]. 3.2 Model Architecture As depicted in Figure 1, TUTTI employs an encoder- decoder architecture based on the Transformer net- work [19], tailored specifically for A2S transcription. The model directly ingests audio spectrograms and au- toregressively generates hierarchical symbolic patches. It is composed of the following key modules: Acoustic Adapter: The encoder’s input consists of continuous acoustic features. Specifically, raw audio is first transformed into a spectrogram representation with the shape of T× F , where T represents the number of time steps and F denotes the frequency bins. To align these high-dimensional acoustic features with the inter- nal working dimension of the Transformer, we apply a dedicated linear projection layer. This layer maps the 1025-dimensional vector at each time step into a dense continuous embedding of dimension 512, matching the model’s hidden size (d model ). This projection acts as a lightweight acoustic embedding mechanism, compressing the frequency information while preserving the tempo- ral sequence length, enabling the Transformer to directly “read” the audio. Acoustic-level Encoder: Operating on the sequence of continuous embeddings produced by the projection layer, the encoder is responsible for extracting high-level acous- tic and harmonic contexts. The encoder captures both local transient events (e.g., note onsets) and global structural de- pendencies (e.g., phrasing and sustained pitches) directly from the spectrogram sequence over the T time steps. Cross-Modal Patch-level Decoder: To handle the highly structured and dense nature of musical scores, we adopt the hierarchical decoding strategy [24], tailoring it for cross-modal alignment. This module takes the em- beddings of previously encoded acoustic features as in- put. This allows the patch-level decoder to dynamically align the global musical structure (e.g., bar-level pacing and polyphony) with corresponding acoustic segments in the audio sequence, ultimately outputting a dense contex- tual representation for the next predicted bar. Character-level Decoder: Operating in a cascaded, step-by-step manner, this module acts as the final translator into discrete ABC notation. Taking the dense contextual representation generated by the patch-level decoder as its condition vector, it autoregressively decodes the specific musical tokens (characters) within that patch. By focusing on local syntactic information—such as specific pitches, durations, and structural symbols—it sequentially recon- structs the exact musical content of each bar patch until the full target score is generated. This hierarchical design ef- fectively decouples the global audio-to-measure alignment from the local measure-to-character generation, ensuring both structural coherence and syntactic correctness. 4. DATASET To overcome the severe scarcity of high-quality, real- world data, we introduce a massive, fully synthesized multi-instrumentation TuttiCorpus dataset. A fundamen- tal contribution of this work is the creation of this dataset from scratch:rather than relying on existing, heav- ily constrained symbolic repositories, we leverage Nota- Gen [25]—a symbolic music generation model—to gen- erate 363,610 unique ABC scores, conditioned on di- verse prompts specifying musical periods (e.g., Baroque, Romantic), composers (e.g., Bach, Beethoven), and in- strumentations (e.g., Keyboard, Chamber). This ensures stylistic variety and musical coherence, as NotaGen’s fine- tuning on high-quality classical corpora and RL align- ment enhance generation quality. To mitigate potential data leakage from NotaGen’s training distribution, we con- ducted a rigorous similarity audit between our generated corpus and the ground-truth test set. We computed the normalized Levenshtein (edit) distance over canonicalized musical sequences to ensure that no generated piece dupli- cates or closely resembles any test sample. Any synthetic instance exceeding a minimal similarity threshold was strictly removed, guaranteeing that the evaluation bench- mark remains completely unseen. 4.1 Data Synthesis and Alignment Pipeline 4.1.1 Score Standardization and Compaction Raw ABC notation files often contain heterogeneous meta- data, varying repeat structures, and empty instrumental voices, which introduce noise during sequence modeling. We first standardize the ABC files by removing redun- dant metadata, stylistic textual annotations, and unplayable symbols. Crucially, we unroll all repeat structures to en- sure the structural linearity of the score matches the tem- poral linearity of the audio sequence. Furthermore, we dy- namically detect and strip empty voices, transferring nec- essary instrumental metadata to active voices, and com- press the multi-line voice structures into a dense, inline sequence representation. This compact format maximizes the efficiency of the Transformer model during training. 4.1.2 Expressive Performance Rendering (EPR) A major challenge in A2S transcription is the discrepancy between rigid, quantized synthetic audio and human ex- pressive variations. To improve model generalization [1], we apply Expressive Performance Rendering (EPR) prior to audio synthesis via three parallel pipelines. A Plain pipeline applies strict quantization with baseline dynam- ics compression and voice balancing. For piano tracks, we utilize VirtuosoNet [28] to generate human-like expressive performances natively. Finally, since neural EPR is largely limited to pianos, we introduce a Rule-based EPR for other instruments that injects controlled perturbations, including velocity humanization (±15), timing jitter (±10 ms), and tempo rubato (±3%). 4.1.3 Audio Synthesis and High-Precision Alignment Instead of relying on standard General MIDI (GM) syn- thesizers, which lack acoustic nuances, we render the pro- cessed MIDI files into high-quality audio using the SFZ sample format via sfizz. 1 We extract the required pitch ranges from the ABC score to dynamically match each voice with the most appropriate acoustic sample library. The resulting individual audio tracks are mixed with linear overlay and normalized to a target loudness of -23.0 LUFS. 1 https://github.com/sfztools/sfizz To establish ground-truth alignment for segmentation, we employ Dual Dynamic Time Warping (DualDTW) [29] via Parangonar 2 to align the expressive MIDI perfor- mances back to the quantized MusicXML scores. This yields precise, bar-level temporal anchors. Using these an- chors, we greedily segment both the continuous audio and the inline ABC scores into aligned chunks no longer than 14.8 seconds. Finally, the segmented audio clips are converted into power spectrograms (sample rate 22,050 Hz, window size 2048, hop length 160) and log-dB normalized, serving as the direct input to our model. 4.2 Data Statistics Throughourautomatedgenerationandsynthesis pipeline, we successfully compiled a massive, multi- instrumentation dataset comprising 363,610 unique and fully aligned data pairs. Our dataset introduces a high level of instrumentation and compositional diversity, making it highly suitable for training generalized transcription models. As detailed in Table 1 and Figure 2, the dataset cov- ers an extensive range of instruments across different fam- ilies, including strings, keyboards, and various wood- winds/brass. In terms of instrumentation configurations, the dataset naturally follows a long-tailed distribution char- acteristic of diverse musical compositions. While solo configurations account for 64.6% of the data—providing a solid foundation for fundamental pitch and duration recognition—a substantial portion of the dataset is ded- icated to complex polyphonic and multi-instrumentation combinations.Specifically, duets and trios constitute nearly 29% of the corpus, and approximately 24,000 in- stances feature quartets, quintets, or larger ensembles. This structural diversity forces the model to learn generalized acoustic representations and robust source identification capabilities, rather than overfitting to the timbre of a sin- gle instrument. 5. EXPERIMENT 5.1 Settings The experiments are designed to systematically assess the capabilities and generalizability of TUTTI in various A2S tasks. We utilize the TuttiCorpus dataset, which is ran- domly split into 95% for training and 5% for testing. TUTTI features a 9-layer patch-level encoder and de- coder, a 3-layer character-level decoder, and a hidden size of 512, amounting to 88.9M parameters. A linear pro- jection layer maps the 1025-dimensional spectrogram fea- tures into the encoder’s input space. The model processes audio inputs of up to approximately 14.8 seconds using a patch length of 2048. It employs a 128-size ASCII-based vocabulary, using characters 0–2 for special tokens (pad, bos, and eos). 2 https://github.com/sildater/parangonar CategoryDetailsCount% Total SegmentsAll audio-score pairs363,610100.00 Top 5Piano228,964– InstrumentsViolin208,119– (by frequency)Cello102,367– Harpsichord63,974– Viola44,912– EnsembleSolo (1 Inst.)235,03264.64 FormatsDuet (2 Inst.)51,04914.04 (by size)Trio (3 Inst.)53,60114.74 Quartet (4 Inst.)14,0103.85 Quintet (5 Inst.)5,5221.52 Large Ensemble4,3961.21 Table 1: Detailed statistics of the TuttiCorpus dataset, highlighting instrumentation frequency and ensemble for- mats. 050000100000150000200000 Number of pieces Solo Trio Duet Quartet Quintet Large Ens. 235,032 53,601 51,049 14,010 5,522 4,396 (a) Orchestration distribution. 050000100000150000200000 Number of pieces Piano Violin Cello Harpsichord Viola Women (Voice) Men (Voice) Oboe Contrabass Flute 228,964 208,119 102,367 63,974 44,912 36,352 31,345 28,826 20,186 16,202 (b) Instrument distribution. Figure 2: Distributions of ensemble types (left) and indi- vidual instruments (right) across the TuttiCorpus dataset. DatasetMethod Metrics (MV2H) Multi-pitchVoiceMeterHarmonyNote ValueTotal ASAP Full Instrumentation Pre-training70.787.590.955.690.776.1 † Single Instrumentation Pre-training66.286.790.657.190.175.0 † Without Pre-training13.274.278.454.671.553.4 † Zeng et al. [1]63.388.4–54.590.774.2 † Quartets Full Instrumentation Pre-training97.799.597.883.499.795.6 Single Instrumentation Pre-training93.898.495.383.499.793.0 Without Pre-training46.285.582.956.092.072.5 Alfaro-Contreras et al. [2]70.692.170.797.693.484.9 Tenor Saxophone Full Instrumentation Pre-training80.4100.096.151.898.085.3 Without Pre-training26.992.385.926.946.855.8 Martínez-Sevilla et al. [3]48.165.261.363.064.560.3 Alto Saxophone Full Instrumentation Pre-training82.8100.096.646.598.784.9 Without Pre-training23.491.779.320.847.452.5 Martínez-Sevilla et al. [3]40.167.357.667.267.159.9 Table 2: Performance comparison (MV2H scores [30]) between TUTTI and baseline models across multiple datasets. Bold values indicate the best performance in each category. † Total score is calculated as the average of four available metrics. The AdamW optimizer [31] is used with a learning rate of 2e-4 for pre-training and 1e-5 for fine-tuning. Pre- training includes a 1000-step warmup followed by a con- stant learning rate over 32 epochs, with a batch size of 10 per GPU. Pre-training was conducted on 8 H800 GPUs, taking approximately 6 days to complete. Fine-tuning was performed on 6 H800 GPUs with a batch size of 6 per GPU over 32 epochs. In ablation studies, we investigate the effects of multi- instrumentation learning on TUTTI, considering three set- tings: 1) omitting pre-training, 2) using only the down- stream task-specific data from TuttiCorpus, or 3) utiliz- ing the entire TuttiCorpus, which includes all instrumen- tations. In terms of comparative evaluations, we select open- source models that excel in their respective domains for benchmarking. TUTTI is fine-tuned on identical datasets to these models, ensuring fairness in comparison. 5.2 Ablation Studies To better understand the contributing factors to our model’s performance, we conduct ablation studies focusing on two main aspects: 1) the necessity of our large-scale syn- thetic pre-training paradigm, and 2) the effectiveness of multi-instrumentation representation learning compared to instrumentation-specific pre-training. As depicted in Table 2, the model trained strictly on the target datasets experiences a severe performance degrada- tion, falling behind the baseline models across all tasks. This aligns with the well-known data-hungry nature of Transformer architectures [17, 32].Our results clearly demonstrate that without the proposed large-scale syn- thetic data pre-training, the Transformer struggles to gener- alize on limited real-world target data. Thus, our synthetic pre-training is the critical enabler that unlocks the high ca- pacity of the Transformer for A2S tasks. While our exper- iments demonstrate that large-scale pre-training unlocks the potential of purely Transformer-based models, we ac- knowledge that this data-centric paradigm is architecture- agnostic; CNN-based hybrid models would likely exhibit parallel performance gains if exposed to the same scale of pre-training. We further investigate whether pre-training on a diverse mixture of instrumentations provides an advantage over pre-training on a matched, single-instrumentation distri- bution. In the “single instrumentation pre-training” set- ting, we restricted the synthetic pre-training data to match the target task (e.g., using only synthetic piano data for the ASAP piano task, and synthetic string quartets for the Quartets dataset). Note that for the saxophone datasets, this ablation is not applicable since saxophones were en- tirely excluded from our pre-training corpus. The results demonstrate that the full-instrumentation pre-trained model outperforms the single-instrumentation pre-trained counterpart.This indicates that multi- instrumentation pre-training is not merely about covering more instrument types, but it fundamentally enhances the model’s underlying representation capability. By learning to disentangle complex, multi-timbral polyphony across diverse instruments, the model learns more robust univer- sal acoustic features and harmonic relationships, which ultimately benefits its performance even on instrument- specific downstream tasks. 5.3 Comparative Evaluations We compare TUTTI, a generalizable A2S model pre- trained on the TuttiCorpus dataset, with several baseline models. The following baseline models have been selected for comparison: • Zeng et al. [1] employ a convolutional recurrent neural network (CRNN) with a customized hierar- chical decoder and the ASAP (piano) [13] dataset as fine-tuning data. • Alfaro-Contreras et al. [2] utilize a CNN with 2- D positional encoding and a Transformer decoder trained on the string quartet dataset. • Martínez-Sevilla et al. [3] use a CRNN trained on human-recorded saxophone data. As the saxophone is absent from the TuttiCorpus instrumentation, this comparison serves to evaluate the model’s transfer learning capability on unseen instruments. To ensure a comprehensive assessment, we adopt the MV2H metric [30], which comprises five sub-metrics: multi-pitch detection, voice separation, metrical align- ment, harmonic detection, and note value accuracy. These values are averaged to produce the total MV2H score. Note that for Zeng et al. [1], the metrical alignment metric is un- available; thus, its total score is calculated by averaging the remaining four components. Table 2 presents the quantitative results.Overall, TUTTI pre-trained with full instrumentation outperforms all baselines, demonstrating superior transcription capabil- ities across both polyphonic and monophonic tasks. On the ASAP dataset, which involves expressive timing and complex polyphony, our method achieves a significant improvement in multi-pitch accuracy, outperforming the baseline by 7.4 points. While yielding competitive results in voice tracking, TUTTI achieves a higher total score, proving its robustness in handling expressive real-world pi- ano recordings. For the Quartets dataset, our model demonstrates sub- stantial margins over the baseline across almost all met- rics, increasing the total score from 84.9 to 95.6. The near- perfect scores in voice (99.5) and note value (99.7) indicate that our approach reliably transcribes dense multi-track ar- rangements. A key strength of TUTTI is its transfer learning capa- bility. On the Saxophone datasets, our model successfully generalizes to these unseen instruments despite their total absence from the TuttiCorpus training set. Notably, since the saxophone is a monophonic instrument, voice accuracy serves as a diagnostic indicator of whether the model cor- rectly identifies its single-voice nature. Failing to reach 100% implies the model erroneously predicts multiple si- multaneous voices—a hallucination likely caused by insuf- ficient acoustic understanding of the instrument. Our fully pre-trained model achieves this theoretical ceiling (100.0) on both datasets, whereas the task-specific baseline and the non-pre-trained variant do not. We attribute this to the diverse multi-timbral representations learned during pre- training: by exposure to a wide variety of instruments and ensembles, TUTTI acquires a more robust acoustic grounding that generalizes to unseen timbres without over- fitting to a single instrument’s characteristics. 6. CONCLUSION We introduced TUTTI, a unified, Transformer encoder- decoder framework that addresses data scarcity and ar- chitectural complexity bottlenecks in generalizable A2S transcription. By pre-training on TuttiCorpus—our large- scale dataset comprising over 363,000 synthetic multi- instrumentation audio-score pairs—we demonstrated that a data-centric paradigm can effectively compensate for the absence of local inductive biases typically provided by CNN acoustic front-ends. Extensive objective evaluations reveal that TUTTI out- performs established task-specific baselines across diverse polyphonic and monophonic tasks, establishing new over- all state-of-the-art results. Crucially, the model exhibits re- markable transfer learning capability, achieving theoretical voice accuracy on unseen real-world instruments, such as the saxophone, strictly through the robust representational power learned from synthetic data. Finally, we release the TuttiCorpus dataset as a rich and highly scalable resource to propel future research in uni- fied A2S transcription. Although TUTTI marks a signifi- cant step forward, extending our mechanism to seamlessly handle unconstrained, full-piece transcriptions remains a promising avenue for future work. 7. ACKNOWLEDGMENTS This work was supported by the following funding source: Special Program of National Natural Science Foundation of China (Grant No. T2341003). 8. REFERENCES [1] W. Zeng, X. He, and Y. Wang, “End-to-end real-world polyphonic piano audio-to-score transcription with hi- erarchical decoding,” in Proceedings of the Thirty- Third International Joint Conference on Artificial In- telligence (IJCAI), 2024, p. 7788–7795. [2] M. Alfaro-Contreras, A. Ríos-Vila, J. J. Valero-Mas, and J. Calvo-Zaragoza, “A transformer approach for polyphonic audio-to-score transcription,” in ICASSP 2024-2024 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP).IEEE, 2024, p. 706–710. [3] J. C. Martínez-Sevilla, M. Alfaro-Contreras, J. J. Valero-Mas, and J. Calvo-Zaragoza, “Insights into end- to-end audio-to-score transcription with real record- ings: A case study with saxophone works,” in Inter- speech 2023, 2023, p. 2793–2797. [4] J. Qian, H. Meng, T. Zheng, P. Zhu, H. Lin, Y. Dai, H. Xie, W. Cao, R. Shang, J. Wu, H. Liu, H. Wen, J. Zhao, Z. Jiang, Y. Chen, S. Yin, M. Tao, J. Wei, L. Xie, and X. Wang, “Soulx-singer: Towards high-quality zero-shot singing voice synthesis,” 2026. [Online]. Available: https://arxiv.org/abs/2602.07803 [5] A. Cogliati, D. Temperley, and Z. Duan, “Transcrib- ing human piano performances into music notation.” in ISMIR, 2016, p. 758–764. [6] E. Nakamura, E. Benetos, K. Yoshii, and S. Dixon, “Towards complete polyphonic music transcription: Integrating multi-pitch detection and rhythm quanti- zation,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, p. 101–105. [7] K. Shibata, E. Nakamura, and K. Yoshii, “Non-local musical statistics as guides for audio-to-score piano transcription,” Information Sciences, vol. 566, p. 262–280, 2021. [8] R. G. C. Carvalho and P. Smaragdis, “Towards end- to-end polyphonic music transcription: Transforming music audio directly to a score,” in 2017 IEEE Work- shop on Applications of Signal Processing to Audio and Acoustics (WASPAA). IEEE, 2017, p. 151–155. [9] M. A. Román, A. Pertusa, and J. Calvo-Zaragoza, “An end-to-end framework for audio-to-score music tran- scription on monophonic excerpts.” in ISMIR, 2018, p. 34–41. [10] M. A. Román, A. Pertusa, and J. Calvo-Zaragoza, “A holistic approach to polyphonic music transcrip- tion with neural networks,” in Proceedings of the 20th International Society for Music Information Retrieval Conference (ISMIR), 2019, p. 731–737. [11] L. Liu, V. Morfi, and E. Benetos, “Joint multi-pitch detection and score transcription for polyphonic pi- ano music,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP). IEEE, 2021, p. 281–285. [12] A. Morsi and X. Serra, “Bottlenecks and solutions for audio to score alignment research,” in Rao P, Murthy H, Srinivasamurthy A, Bittner R, Caro Repetto R, Goto M, Serra X, Miron M, editors. Proceedings of the 23nd International Society for Music Information Retrieval Conference (ISMIR 2022); 2022 Dec 4-8; Bengaluru, India.[Canada]: International Society for Music Infor- mation Retrieval; 2022. p. 272-9.International Soci- ety for Music Information Retrieval (ISMIR), 2022. [13] F. Foscarin, A. McLeod, P. Rigaux, F. Jacquemard, and M. Sakai, “ASAP: a dataset of aligned scores and per- formances for piano transcription,” in International So- ciety for Music Information Retrieval Conference (IS- MIR), 2020, p. 534–541. [14] W. Chen, J. Keast, J. Moody, C. Moriarty, F. Villalo- bos, V. Winter, X. Zhang, X. Lyu, E. Freeman, J. Wang, S. Cai, and K. M. Kinnaird, “Data usage in mir: His- tory & future recommendations,” in Proceedings of the 20th International Society for Music Information Re- trieval Conference (ISMIR), Delft, Netherlands, 2019, p. 11–18. [15] V. Arroyo, J. J. Valero-Mas, J. Calvo-Zaragoza, and A. Pertusa, “Neural audio-to-score music transcrip- tion for unconstrained polyphony using compact out- put representations,” in ICASSP 2022-2022 IEEE In- ternational Conference on Acoustics, Speech and Sig- nal Processing (ICASSP).IEEE, 2022, p. 4603– 4607. [16] S. d’Ascoli, H. Touvron, M. L. Leavitt, A. S. Mor- cos, G. Biroli, and L. Sagun, “Convit: Improving vi- sion transformers with soft convolutional inductive bi- ases,” in International conference on machine learn- ing. PMLR, 2021, p. 2286–2296. [17] R. Shao and X.-J. Bi, “Transformers meet small datasets,” IEEE Access, vol. 10, p. 118 454–118 464, 2022. [18] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” Advances in neural information processing systems, vol. 27, 2014. [19] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural infor- mation processing systems, vol. 30, 2017. [20] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, p. 9, 2019. [21] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Explor- ing the limits of transfer learning with a unified text- to-text transformer,” Journal of machine learning re- search, vol. 21, no. 140, p. 1–67, 2020. [22] A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan et al., “Deepseek-v3 technical report,” arXiv preprint arXiv:2412.19437, 2024. [23] D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi et al., “Deepseek- r1: Incentivizing reasoning capability in llms via rein- forcement learning,” arXiv preprint arXiv:2501.12948, 2025. [24] S. Wu, Y. Wang, X. Li, F. Yu, and M. Sun, “Melodyt5: A unified score-to-score transformer for symbolic mu- sic processing,” in Proceedings of the 25th Interna- tional Society for Music Information Retrieval Confer- ence (ISMIR), 2024. [25] Y. Wang, S. Wu, J. Hu, X. Du, Y. Peng, Y. Huang, S. Fan, X. Li, F. Yu, and M. Sun, “Notagen: Advanc- ing musicality in symbolic music generation with large language model training paradigms,” in Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence (IJCAI), 2025, p. 10 207– 10 215. [26] S. Wu, X. Li, F. Yu, and M. Sun, “Tunesformer: Forming irish tunes with control codes by bar patch- ing,” in Proceedings of the 2nd Workshop on Human- Centric Music Information Retrieval 2023 co-located with the 24th International Society for Music Infor- mation Retrieval Conference (ISMIR 2023), Milan, Italy,November 10, 2023, 2023. [27] S. Wu, D. Yu, X. Tan, and M. Sun, “Clamp: Con- trastive language-music pre-training for cross-modal symbolic music information retrieval,” in Proceedings of the 24th International Society for Music Informa- tion Retrieval Conference, ISMIR 2023, Milan, Italy, November 5-9, 2023, 2023. [28] D. Jeong, T. Kwon, Y. Kim, K. Lee, and J. Nam, “Vir- tuosonet: A hierarchical rnn-based system for model- ing expressive piano performance.” in ISMIR, 2019, p. 908–915. [29] S. D. Peter, “Online symbolic music alignment with offline reinforcement learning,” in Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), 2023. [30] A. McLeod and M. Steedman, “Evaluating automatic polyphonic music transcription,” in International So- ciety for Music Information Retrieval Conference (IS- MIR), 2018, p. 42–49. [31] I. Loshchilov and F. Hutter, “Decoupled weight de- cay regularization,” in 7th International Conference on Learning Representations, ICLR 2019,New Orleans, LA, USA, May 6-9, 2019, 2017. [32] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language mod- els,” arXiv preprint arXiv:2001.08361, 2020.