Paper deep dive
Rethinking Procedural Audio Pre-training: Source Scaling and Objective Adaptation
Jiajun Peng, Fengrui Liu, Xinyu Liu, Feng Liu
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Procedural audio has emerged as a viable source for transferable audio representation learning, but its design principles remain this http URL revisit two questions: how a procedural source should be scaled, and whether training choices developed on natural audio should transfer unchanged to procedural this http URL a controlled source, we separate scale into formula-class coverage C and within-class rendering diversity this http URL with FDSL and AudioMAE show that these two forms of scale provide different benefits and depend on the learning formulation and downstream task. A matched AudioMAE study further shows that procedural audio favors low mask ratios (10%--25%), whereas AudioSet-28K favors 50%--75%. Shared-codebook analysis reveals lower patch diversity and stronger temporal predictability in procedural audio. These results motivate source-aware procedural pre-training, where source scaling and learning configuration are considered this http URL is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2609.15067v1
- Canonical: https://arxiv.org/abs/2609.15067v1
Trouble viewing inline? Open PDF directly →
Full Text
16,353 characters extracted from source content.
Expand or collapse full text
RETHINKING PROCEDURAL AUDIO PRE-TRAINING: SOURCE SCALING AND OBJECTIVE ADAPTATION Jiajun Peng 1,∗ , Fengrui Liu 2,∗,‡ , Xinyu Liu 1 , Feng Liu 3,† , Senior Member, IEEE 1 School of Data Science, University of Science and Technology of China, Hefei 230026, China 2 School of Computer Science and Technology, East China Normal University, Shanghai 200062, China 3 School of Psychology, Shanghai Jiao Tong University, Shanghai 200030, China ∗ Equal contribution. † Corresponding author. ‡ Project lead. ABSTRACT Procedural audio has emerged as a viable source for transfer- able audio representation learning, but its design principles remain unclear.We revisit two questions: how a procedu- ral source should be scaled, and whether training choices developed on natural audio should transfer unchanged to procedural data.Using a controlled source, we separate scale into formula-class coverage C and within-class rendering di- versity I .Experiments with FDSL and AudioMAE show that these two forms of scale provide different benefits and depend on the learning formulation and downstream task. A matched AudioMAE study further shows that procedural audio favors low mask ratios (10–25%), whereas AudioSet-28K favors 50–75%.Shared-codebook analysis reveals lower patch diversity and stronger temporal predictability in procedural audio. These results motivate source-aware procedural pre- training, where source scaling and learning configuration are considered jointly.Code is available at https://github. com/Cross-Innovation-Lab/Formula-Bank. Index Terms— procedural audio, synthetic audio, audio pre-training, masked autoencoders 1. INTRODUCTION Formula-driven and procedural generation provide an alterna- tive route to representation learning without relying entirely on recorded or human-annotated data. FDSL showed that representations can be learned from patterns and labels gen- erated by mathematical formulas [1], while recent work ex- tended this idea to audio, showing that synthetic and proce- dural signals can support transferable representations [2, 3] and complement real recordings [4]. These studies establish the feasibility of procedural audio for representation learning. We therefore move from asking whether procedural audio can work to how procedural pre-training should be designed. Unlike recorded datasets, procedural generation explicitly exposes how a source grows. Increasing the number of sam- ples can introduce new generative structures or more acoustic Controlled source: FormulaBank I 1 I 2 ... I n C 1 C 2 ⋮ C C ⋯ ⋯ ⋯ ⋮ Formula classes (C) Renderings Instances (I) Shared pre-training pipeline Audio clip Log-Mel spectrogram 16×16 patches Random masking MAE pre-training x(t) Frequency Time Masked Visible ⋯ Encoder ⋯ Decoder ⋯ Reconstruct masked patches From controlled data to pre-training 푥 푐,푖 =퐺휙 푐 ,휈 푐,푖 Formula classes (C) ⋮ ⋮ Rendered instances (I) ⋯ N(C,I) = C × I clips Fig. 1. Overview of the study. realizations of existing ones, which we denote as formula- class coverage C and within-class rendering diversity I . Al- though both increase source size, they need not provide the same learning signal. This raises our first question: should procedural scale be described by sample count alone, or by distinct structural and rendering dimensions? The value of these dimensions may also depend on the learning formulation. As we show 1, FDSL and Au- dioMAE [5] respond differently to changes in C and I , suggesting an interaction between source composition and the learner. This leads to our second question: should train- ing choices developed on natural audio remain unchanged when the source changes? This is particularly relevant to masked audio pre-training, whose masking strategies are largely developed on natural recordings [5, 6, 7]. To study these questions, we construct FormulaBank, where C and I can be varied independently. We examine the two scaling axes under FDSL and AudioMAE with matched- data and matched-compute controls, and test the same source across heterogeneous pre-training paradigms [5, 8, 9, 10]. Finally, we conduct a matched FormulaBank–AudioSet- 28K [11] study that holds the model, data size, optimization, mask-ratio grid, and downstream evaluation fixed while changing only the pre-training source. The results show that Submitted to ICASSP 2027 arXiv:2609.15067v1 [cs.SD] 14 Sep 2026 Fig. 2. Controlled procedural source and pre-training pipeline. FormulaBank separates formula-class coverage C from render- ing diversity I , with N(C,I) = C× I clips. procedural scale is multidimensional and learner-dependent, while the preferred masking regime is strongly source- dependent. Our main contributions are: • We disentangle procedural source scale into formula-class coverage and within-class rendering diversity, and show that their relative benefits depend on the learning formulation and downstream task. • We show that the same procedural source supports heteroge- neous audio pre-training paradigms, indicating that its utility is not tied to a single learning objective. • A matched AudioMAE study reveals substantially differ- ent preferred masking regimes for procedural audio and AudioSet-28K; shared-codebook statistics further show lower patch diversity and stronger temporal predictability in procedural audio. 2. METHODOLOGY 2.1. Controlled Procedural Source We construct FormulaBank from the AudioPG generator [3] to separate two forms of procedural source diversity. 2 Let x c,i = G(φ c ,ν c,i ),(1) where φ c denotes the generative structure of formula class c, and ν c,i denotes the rendering variables of its i-th instance. Samples within the same class therefore share the same un- derlying structure while differing in acoustic realization. With C formula classes and I renderings per class, the source contains N(C,I) = CI(2) clips. We use C to represent structural coverage and I to rep- resent within-structure rendering diversity. This construction allows the two forms of procedural scale to be varied inde- pendently. 2.2. Scaling Procedural Sources We study the two scaling axes separately: I = 125, C ∈14, 28, 56, 112, 224,(3) C = 224, I ∈16, 32, 64, 125, 250, 500, 1000.(4) The same source configurations are evaluated under FDSL [1] and AudioMAE [5], allowing us to examine whether the value of structural coverage and rendering diver- sity depends on the learning formulation. We additionally use fixed-data and fixed-compute controls to separate changes in source composition from additional training exposure. 2.3. Generality and Source Adaptation We first test whether the procedural source transfers across different representation-learning paradigms by pre-training AudioMAE [5], BYOL-A [8], SSAST [9], and PaSST [10] on the same FormulaBank source. 2 Table 1. Scaling formula-class coverage and rendering diversity under FDSL and AudioMAE pre-training. Results within each pre-training setting use the same downstream evaluation protocol. FDSLAudioMAE AxisScaleESC-50 Acc. ↑ US8K Acc. ↑ FSD50K mAP↑ SCv2 Acc. ↑ AS-20K mAP↑ESC-50 Acc. ↑ US8K Acc. ↑ FSD50K mAP↑ SCv2 Acc. ↑ AS-20K mAP↑ Formula-class coverage (I = 125) C = 1476.5076.110.450797.300.133071.7578.610.443196.790.1163 C = 2874.5077.540.459997.080.138276.2581.120.485496.660.1579 C = 5676.0075.390.472097.420.131078.5084.590.498197.210.1657 C = 11276.2580.760.495797.230.136677.5083.750.492197.190.1746 C = 22477.0082.920.491997.340.142377.5083.990.507397.160.1895 Rendering diversity (C = 224) I = 16 76.7576.220.474297.400.132966.5075.510.436596.520.1238 I = 3276.2581.360.478597.220.137372.7580.050.473297.040.1549 I = 6475.5081.000.480597.060.142477.7584.110.490197.220.1680 I = 125 77.0082.920.491997.340.142377.5083.990.507397.160.1895 I = 25077.2581.720.486597.180.150682.2583.990.522197.420.1966 I = 50078.5080.760.492897.270.153581.2585.900.530397.270.2017 I = 1000 77.7579.690.486897.070.148082.7585.900.541397.230.2006 We then isolate the effect of source domain using a matched AudioMAE setup. FormulaBank and a 28K-clip AudioSet subset [11] use the same model, optimization, input representation, and downstream evaluation, while only the pre-training source is changed. For both sources, we sweep r ∈.10,.25,.50,.75,.80,.90,(5) where r is the fraction of masked spectrogram patches. 2.4. Source Statistics To relate masking behavior to source statistics, we quantize patches from FormulaBank and AudioSet-28K using a shared codebook. Let Z denote the resulting discrete patch code. We measure patch diversity by H(Z) =− X k p(k) log 2 p(k),(6) and local temporal predictability by H t = H(Z t+1,f | Z t,f ).(7) Lower H t indicates stronger predictability between adjacent time patches. 3. EXPERIMENTS 3.1. Experimental Setup Datasets and metrics.Pre-training uses FormulaBank as the procedural source and AudioSet-28K[11] as the matched natural-audio control where applicable.Down- stream evaluation is conducted on ESC-50 [12], Urban- Sound8K (US8K) [13], FSD50K [14], Speech Commands v2 (SCv2), and AudioSet-20K (AS-20K). We report classi- fication accuracy for ESC-50, US8K, and SCv2, and mean average precision (mAP) for FSD50K and AS-20K. Implementation details. AudioMAE experiments use 128-bin log-Mel spectrograms of size 1024× 128 with non- overlapping 16× 16 time–frequency patches. FormulaBank scaling and masking experiments use ViT-S, while the cross- method study retains each method’s native encoder and train- ing recipe. All downstream models are fully fine-tuned using the same protocol for a given downstream dataset. 3.2. Multidimensional Procedural Scaling We first examine how procedural pre-training scales along two controllable source dimensions: formula-class coverage C and within-class rendering diversity I . We fix I = 125 when varying C, and fix C = 224 when varying I , under both FDSL [1] and AudioMAE [5]. Table 1 shows that the two forms of scale are not equiva- lent. Under FDSL, broader formula-class coverage produces the clearer trend, whereas rendering diversity yields smaller and less consistent changes. AudioMAE shows a different response: coverage gives rapid early gains followed by task-dependent saturation, while rendering diversity provides broader improvements across downstream tasks. To separate source composition from additional training exposure, we use matched-budget controls: FDSL fixes the source size to 28K clips, while AudioMAE fixes optimization exposure to 7M clip presentations. Figure 3 shows that these trends persist under controlled budgets, supporting a multidimensional view of procedural scale whose axes depend on both the learning formulation and downstream task. 3.3. Generality Across Learning Paradigms We next test whether the procedural source is tied to a par- ticular pre-training objective. The same FormulaBank source is used with AudioMAE [5], BYOL-A [8], SSAST [9], and PaSST [10], while retaining each method’s native encoder and training recipe. Table 2 shows that all four learning paradigms produce transferable representations from the same procedural source. Since the architectures and native training recipes differ, this 3 Fig. 3. Controlled-budget scaling trends. Left: FDSL with 28K unique clips, varying source composition from 56× 500 to 224× 125. Middle: AudioMAE with 7M clip presentations, increasing formula-class coverage at I = 125. Right: AudioMAE under the same compute budget, increasing rendering diversity at C = 224. Scores are normalized within each downstream task. .10.25.50.75.80.90 mask ratio 78 80 82 84 86 88 ESC-50 Acc. (%) Synthetic AudioSet-28K .10.25.50.75.80.90 mask ratio 82 84 86 88 90 US8K Acc. (%) .10.25.50.75.80.90 mask ratio 0.50 0.52 0.54 0.56 0.58 0.60 FSD50K mAP Fig. 4. Source-dependent masking under full fine-tuning. FormulaBank favors low masking (.10–.25), whereas the matched AudioSet-28K control favors moderate-to-high masking (.50–.75). The AudioMAE architecture, clip count, mask-ratio grid, and downstream evaluation are held fixed across the two sources. Table 2.Cross-method transfer from FormulaBank pre- training. Each method uses its native configuration and is evaluated by full fine-tuning. MethodEncoderESC-50 US8K FSD50K SCv2 AS-20K AudioMAEViT-B86.2585.78.564997.77 .241169 BYOL-AAudioNTT2022-CNN75.0076.82.466096.70 .152646 SSASTViT-B81.0081.36.516796.72 .199608 PaSST-noImageNetViT-B75.5077.66.515696.81 .179001 ScratchViT-B69.5073.36.409096.74 .136188 experiment evaluates source compatibility rather than ranking the pre-training methods. 3.4. Source-Dependent Masking and Source Statistics We finally ask whether the learning configuration should adapt when the pre-training source changes. FormulaBank and a matched 28K-clip AudioSet subset [11] are compared under the same AudioMAE setup, with architecture, opti- mization, mask-ratio grid, and downstream evaluation held fixed. Figure 4 shows a clear source-dependent shift. Formula- Bank performs best at low mask ratios (r = .10–.25), whereas AudioSet-28K favors r = .50–.75. This trend is consistent across downstream tasks and repeated runs. To characterize Table 3. Shared-codebook patch entropy in bits. Higher val- ues indicate greater patch diversity or lower short-range pre- dictability. SourceMarginal H Within-clip H Temporal H t FormulaBank3.1402.104.715 AudioSet-28K6.7014.9413.807 this difference, we compare the two sources in a shared dis- crete patch space.Table 3 shows that FormulaBank has lower patch diversity and substantially stronger local temporal pre- dictability than AudioSet-28K. This suggests that the same masking ratio induces different prediction difficulty across source domains, motivating source-dependent masking. 4. CONCLUSION Overall, these results motivate source–objective co-design: masking should be adapted to the statistics of the pre-training source rather than inherited as a source-independent default. 4 5. REFERENCES [1] Hirokatsu Kataoka, Kazushige Okayasu, Asato Mat- sumoto, Eisuke Yamagata, Ryosuke Yamada, Nakamasa Inoue, Akio Nakamura, and Yutaka Satoh, “Pre-training without natural images,” International Journal of Com- puter Vision, vol. 130, p. 990–1007, 2022. [2] Yuchi Ishikawa, Tatsuya Komatsu, and Yoshimitsu Aoki, “Pre-training with synthetic patterns for audio,” in Proceedings of ICASSP, 2025. [3] Fengrui Liu, Ruiyang Huang, Qijian Zheng, Yuanfang Wang, and Feng Liu, From Physics to Representation: Audio Learning with Synthetic Pre-training via Proce- dural Generation, p. 595–604, Association for Com- puting Machinery, New York, NY, USA, 2026. [4] Fengrui Liu, Ruiyang Huang, Qijian Zheng, Yuanfang Wang, and Feng Liu, “Prp: Procedural-to-real masked pre-training for transferable and interpretable audio rep- resentations,” IEEE Transactions on Audio, Speech and Language Processing, vol. 34, p. 4074–4083, 2026. [5] Po-Yao Huang et al., “Masked autoencoders that listen,” in Advances in Neural Information Processing Systems, 2022. [6] Dading Chong, Helin Wang, Peilin Zhou, and Qingcheng Zeng, “Masked spectrogram prediction for self-supervised audio pre-training,” in Proceedings of ICASSP, 2023. [7] Alan Baade, Puyuan Peng, and David Harwath, “Mae- ast: Masked autoencoding audio spectrogram trans- former,” in Proceedings of Interspeech, 2022. [8] Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, et al., “Byol for audio: Self-supervised learning for general-purpose audio representation,” in Proceedings of IJCNN, 2021. [9] Yuan Gong, Yu-An Chung, and James Glass, “Ssast: Self-supervised audio spectrogram transformer,” in Pro- ceedings of AAAI, 2022. [10] Khaled Koutini, Hamid Eghbal-zadeh, Matthias Dorfer, et al., “Efficient training of audio transformers with patchout,” in Proceedings of Interspeech, 2022. [11] Jort F. Gemmeke et al., “Audio set: An ontology and human-labeled dataset for audio events,” in Proceedings of ICASSP, 2017. [12] Karol J. Piczak, “Esc: Dataset for environmental sound classification,”in Proceedings of ACM Multimedia, 2015. [13] Justin Salamon, Christopher Jacoby, and Juan Pablo Bello, “A dataset and taxonomy for urban sound re- search,” in Proceedings of ACM Multimedia, 2014. [14] Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra, “Fsd50k: An open dataset of human-labeled sound events,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, p. 829–852, 2022. 5