Paper deep dive
MSR-HuBERT: Self-supervised Pre-training for Adaptation to Multiple Sampling Rates
Zikang Huang, Meng Ge, Tianrui Wang, Xuanchen Li, Xiaobao Wang, Longbiao Wang, Jianwu Dang
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/26/2026, 1:43:40 AM
Summary
MSR-HuBERT is a novel self-supervised learning framework designed to address the 'resolution mismatch' problem in speech processing, where fixed-rate models struggle with multi-sampling-rate data. By replacing the standard single-rate downsampling CNN with a multi-sampling-rate adaptive downsampling CNN, MSR-HuBERT maps raw waveforms of varying sampling rates to a shared temporal resolution without resampling. This approach preserves high-frequency details for speech reconstruction while maintaining low-frequency semantic structure for speech recognition, all while remaining compatible with existing HuBERT-based architectures and improvements.
Entities (5)
Relation Signals (3)
MSR-HuBERT → isbasedon → HuBERT
confidence 100% · Building on HuBERT, we replace its single-rate downsampling CNN
MSR-HuBERT → solves → Resolution Mismatch Problem
confidence 95% · MSRHuBERT is a speech SSL framework that processes speech sampled at multiple sampling rates and is explicitly designed to resolve the resolution mismatch problem.
MSR-HuBERT → uses → Transformer Encoder
confidence 95% · MSRHuBERT retains HuBERT's mask-prediction objective and Transformer encoder
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Self-supervised learning (SSL) has advanced speech processing. However, existing speech SSL methods typically assume a single sampling rate and struggle with mixed-rate data due to temporal resolution mismatch. To address this limitation, we propose MSRHuBERT, a multi-sampling-rate adaptive pre-training method. Building on HuBERT, we replace its single-rate downsampling CNN with a multi-sampling-rate adaptive downsampling CNN that maps raw waveforms from different sampling rates to a shared temporal resolution without resampling. This design enables unified mixed-rate pre-training and fine-tuning. In experiments spanning 16 to 48 kHz, MSRHuBERT outperforms HuBERT on speech recognition and full-band speech reconstruction, preserving high-frequency detail while modeling low-frequency semantic structure. Moreover, MSRHuBERT retains HuBERT's mask-prediction objective and Transformer encoder, so existing analyses and improvements that were developed for HuBERT can apply directly.
Tags
Links
- Source: https://arxiv.org/abs/2603.23048v1
- Canonical: https://arxiv.org/abs/2603.23048v1
Trouble viewing inline? Open PDF directly →
Full Text
28,617 characters extracted from source content.
Expand or collapse full text
MSR-HuBERT: Self-supervised Pre-training for Adaptation to Multiple Sampling Rates Zikang Huang 1 , Meng Ge 1 , Tianrui Wang 1 , Xuanchen Li 1 , Xiaobao Wang 1 , Longbiao Wang 1,2,∗ , Jianwu Dang 3 1 Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, China 2 Huiyan Technology Company, Ltd., China 3 Chinese Academy of Sciences, China huangzikang@tju.edu.cn Abstract Self-supervised learning (SSL) has advanced speech process- ing.However, existing speech SSL methods typically as- sume a single sampling rate and struggle with mixed-rate data due to temporal resolution mismatch. To address this limita- tion, we propose MSRHuBERT, a multi-sampling-rate adap- tive pre-training method. Building on HuBERT, we replace its single-rate downsampling CNN with a multi-sampling-rate adaptive downsampling CNN that maps raw waveforms from different sampling rates to a shared temporal resolution with- out resampling. This design enables unified mixed-rate pre- training and fine-tuning. In experiments spanning 16 to 48 kHz, MSRHuBERT outperforms HuBERT on speech recognition and full-band speech reconstruction, preserving high-frequency detail while modeling low-frequency semantic structure. More- over, MSRHuBERT retains HuBERT’s mask-prediction objec- tive and Transformer encoder, so existing analyses and improve- ments that were developed for HuBERT can apply directly. Index Terms: self-supervised learning, sampling-rate adapta- tion, speech reconstruction, speech recognition, fine-tuning 1. Introduction Self-supervised learning (SSL) has enabled major advances in natural language processing and computer vision [1, 2]. In speech, SSL has emerged as a leading approach for learning powerful speech representations by pre-training on large unla- beled corpora using objectives that derive supervision from the signal itself rather than from human annotations [3, 4, 5, 6]. After pre-training, these models act as representation extractors and are fine-tuned on labeled data for downstream tasks such as automatic speech recognition (ASR) [7, 8, 9]. Mainstream speech SSL models (e.g., wav2vec 2.0, Hu- BERT, and WavLM) adopt a common design: a convolutional encoder compresses the raw waveform into frame-level features with a fixed 20 ms frame shift, which serve as the basic units for pre-training and are realized by a 320× temporal downsampling for 16 kHz audio [10, 11, 12]. Consequently, these models are effectively tied to 16 kHz. For other sampling rates, the ex- tracted frame-level features’ frame shift and temporal resolu- tion no longer match the expected 20 ms. This resolution mis- match leads the models to fail to function correctly during both pre-training and fine-tuning. We refer to this issue as the res- olution mismatch problem, as illustrated in Fig. 1. The prob- lem prevents direct use of non-16 kHz audio in pre-training, reduces data diversity, and limits applicability to downstream ** indicates the corresponding author. 320x Downsampling CNN Extractor 16k Hz24k Hz48k Hz 25ms20ms16.66ms13.33ms8.33ms6.67ms Figure 1: The resolution mismatch across diverse sampling rates with a fixed 320× downsampling CNN. tasks that operate at other sampling rates [13, 14, 15]. Address- ing sampling-rate compatibility is therefore crucial for robust and widely usable speech SSL. To our knowledge, this work represents a pioneering effort to explicitly explore solutions to this resolution mismatch in speech SSL. Simple remedies are unsatisfactory: training separate mod- els per sampling rate is costly, and resampling high-rate au- dio to 16 kHz discards high-frequency information that is use- ful for downstream performance [16]. Frequency-domain tech- niques (e.g., subband decomposition or fixed-duration STFTs) can achieve sampling-rate independence in speech enhance- ment [17, 18], but they do not integrate naturally into exist- ing SSL pipelines without disrupting the original paradigm, be- cause most speech SSL models operate on the time-domain, and the few studies on frequency-domain adaptation target improv- ing training efficiency rather than performance [19]. In this paper, we propose MSRHuBERT, a multi-sampling- rate adaptive pre-training method. By introducing a multi- sampling-rate adaptive downsampling convolutional feature ex- tractor, routing audio of different sampling rates through a con- volutional encoder with rate-specific downsampling, it maps in- puts to frame-level features on a common temporal resolution, thereby alleviating the resolution mismatch problem. Mean- while, we retain the standard SSL paradigm, the training ob- jective and architecture. We implement our design on a Hu- BERT backbone and conduct pre-training and downstream fine- tuning on datasets sampled at multiple rates. Results show our approach achieves comparable performance on the ASR task that depends on low-frequency signal content, while im- proving performance on the full-band speech reconstruction task that relies on high-frequency information. Moreover, be- cause we retain the original SSL objective and architecture, im- provements developed for HuBERT can be incorporated into our method to yield further gains. Finally, as the first work to explicitly mitigate multi-sampling-rate resolution mismatch in speech SSL, we empirically quantify the severe degrada- arXiv:2603.23048v1 [cs.SD] 24 Mar 2026 tion that mismatch induces on ASR and speech reconstruction. Our main contributions are: (1) MSRHuBERT, a speech SSL framework adaptive for multiple sampling rates, including a multi-sampling-rate adaptive downsampling convolutional ex- tractor that avoids resolution mismatch while preserving the core SSL paradigm; (2) empirical evidence across multiple sam- pling rates that the learned representations by our method retain low-frequency content that is important for ASR while also cap- turing high-frequency details beneficial for reconstruction; (3) the first identification and formalization of the resolution mis- match problem in speech SSL. 2. HuBERT HuBERT [11] is a typical speech SSL framework. It benefits from an offline clustering (e.g., k-means) to generate frame- level pseudo labels U = [u 1 ,u 2 ,...,u T ], where T is the num- ber of speech frames. A fixed 320x downsampling convolu- tional neural network (CNN) F(·) converts 16 kHz speech s 16k into the frame-level feature H = [h 1 ,h 2 ,...,h T ] with a frame shift of 20 ms, which is then fed to a Transformer encoder G(·) to get the contextual representationC = [c 1 ,c 2 ,...,c T ]. During pre-training, the frame-level features H are masked randomly before they are passed to the Transformer encoder. The whole pipeline can be formalized as C = [c 1 ,c 2 ,...,c T ] = G(M(F(s 16k ))),(1) where M(·) is the mask operation on frame-level features. Fi- nally, the model is trained to predict the pseudo labels of the masked frames using a cross-entropy (CE) loss function L CE (C,U) =− X t∈O logp(u t |c t ),(2) where O denotes the set of indices masked in H . Note that, when applied to downstream tasks, H is no longer masked and the obtained unmasked contextual representation C is used. 3. Proposed Method Fig. 2 illustrates the overall architecture of MSRHuBERT. MSRHuBERT is a speech SSL framework that processes speech sampled at multiple sampling rates and is explicitly designed to resolve the resolution mismatch problem. Given speeches at diverse sampling rates, the model maps them to a common temporal resolution and performs mixed-rate pre- training using a shared codebook. MSRHuBERT departs from HuBERT in a principal respect: a multi-sampling-rate adaptive downsampling CNN, which aligns frame-level features across sampling rates, enabling direct pre-training on raw multi-rate waveforms while preserving the intended 20 ms frame shift, modeling both low- and high-frequency information. 3.1. Multi-sampling-rate Adaptive Downsampling CNN The multi-sampling-rate adaptive downsampling CNN is de- signed to compress speech signals recorded at different sam- pling rates into frame-level features that share a common tem- poral resolution without resampling, thereby tackling the reso- lution mismatch problem while preserving high-frequency in- formation. Specifically, given an input waveform s ak sampled at a kHz, the model routes it to a rate-specific downsampling CNN F dr (·) with downsampling ratio dr so that the resulting frame-level features match HuBERT’s temporal convention of Multi-Sampling-Rate Adaptive Downsampling CNN 16k Hz24k Hz48k Hz 25ms20ms25ms20ms25ms20ms 320x downsample480x downsample960x downsample MASKMASKMASK Transformer Speech Tokenizer (Kmeans) Figure 2: The architecture of MSRHuBERT, which takes raw waveforms at different sampling rates as input. The multi- sampling-rate adaptive downsampling CNN can map wave- forms to a common temporal resolution by designing rate- specific downsampling, and supports mixed-rate pre-training. 20 ms frame shift. Inspired by [20], we append a layer normal- ization after each downsampling CNN, which removes feature aggregation inconsistency across rate-specific CNNs and maps all extracted features into a shared feature space by normalizing mean and variance independently per branch [21]. H = [h 1 ,h 2 ,...,h T ] = LN dr F dr (s ak ) ,(3) where LN dr denotes layer normalizing attached to F dr , and downsampling ratiodr = ak×0.02, achieved through carefully designing the stride and kernel width of each F dr (detailed in Table 1). This design ensures that all sampling rates are aligned to the same temporal resolution before being fed to the shared Transformer encoder. Compared to HuBERT, the adaptive downsampling CNN enables direct processing of raw high-sampling-rate speech without resampling, thereby preserving high-frequency infor- mation, increasing the diversity of pre-training data, and extend- ing applicability to downstream tasks operating at multiple sam- pling rates. Because the extracted features all share the standard temporal scale and lie in a unified feature space, the following mask prediction and Transformer encoder can be retained. 3.2. Mixed-rate Mask Prediction via Single Codebook Because the multi-sampling-rate adaptive downsampling CNN maps inputs from different sampling rates to frame-level fea- tures on a common temporal grid. This unified temporal scale enables the subsequent use of a shared Transformer encoder and a single shared codebook for consistent contextual mod- eling. This unified formulation preserves the original training paradigm while allowing the model to exploit the greater di- versity of multi-rate pre-training data. Furthermore, this de- sign ensures training efficiency, minimizes additional model parameters, and facilitates the effective extraction of robust speech representations. Concretely, mask prediction is per- formed on features with a 20 ms frame shift, a convention used in prior work to yield effective speech representations. Follow- ing the HuBERT paradigm, the frame-level features, produced by the adaptive downsampling branch, are masked randomly and passed to a shared Transformer encoder whose contextual outputs are trained to predict offline clustering pseudo labels, as L =− 1 T X t∈O log exp(sim(Ac t ,e u )/τ) P U u ′ exp(sim(Ac t ,e u ′ )/τ) (4) where A is a projection matrix, e u is the embedding for pseudo label u of frame t, T is the number of masked frames, and τ scales the logit which is set to 0.1. This mixed-rate single shared codebook training enables the model to absorb low- and high-frequency cues present across different sampling rates, and constrain extracted contex- tual representations from diverse rates to lie in a common fea- ture space, yielding rate-invariant representations that simplify fine-tuning on mixed-rate downstream tasks. 4. Experiments 4.1. Pre-training Setup For multi-sampling-rate pre-training, we retain the 960-hour LibriSpeech corpus [22] at 16 kHz used by HuBERT Base to ensure a fair comparison. For the other sampling rates we use the clean subset of the DNS Challenge 2022 dataset [23], origi- nally recorded at 48 kHz. This subset is uniformly resampled to 22.05 kHz, 24 kHz and 48 kHz, producing roughly 193 hours of speech for each rate. All pre-training experiments are implemented using the Fairseq toolkit [24]. We train all models from random ini- tialization [25] for 400k steps on four 24GB NVIDIA 4090D GPUs, using a batch size of 87.5 seconds of audio per GPU and eight times gradient accumulation to match HuBERT’s ef- fective batch configuration. During pre-training, we control same sampling-rate in each batch but mix batches from different sampling-rates within each update because of distributed train- ing and gradient accumulation, allowing efficient learning from diverse data with minimal extra forward computation. Regard- ing parameters, our proposed CNN branches contribute little to the parameter count, which is mainly from the Transformer En- coder. Quantitatively, adding one sampling rate increases pa- rameters by only 3%, and the time cost of mixed training with 4 rates is just 2.6% longer than single-rate training. Other hy- perparameters for the model are kept the same as in [11]. We use the HuBERT base as the baseline and backbone model, and the architecture of our proposed multi-sampling-rate adaptive downsampling CNN is shown in Table 1. Note that HuBERT Base baseline is pre-trained for one sampling rate using the cor- responding single-rate CNN. 4.2. Fine-tuning Setup To evaluate the generality of representations learned by MSRHuBERT across different sampling rates, we perform fine-tuning experiments under the SUPERB evaluation proto- col [26]. In this protocol, the pre-trained model is kept frozen and only the lightweight downstream module is learnable. For each sampling rate, we select a corresponding dataset: train-clean-100 and test-clean subsets of LibriSpeech for 16 kHz [22], LJSpeech for 22.05 kHz [27], train-clean-100 and test-clean subsets of LibriTTS for 24 kHz [28], and VCTK Table 1: Architecture design for proposed multi-sampling-rate adaptive downsampling CNN. For the CNN branch of each sam- pling rate, the stride and kernel width is determined based on the calculation formulas for the downsampling rate, stride, and kernel width, as well as the prime factor decomposition. Sampling RateStridesKernel width 16 kHz5, 2, 2, 2, 2, 2, 210, 3, 3, 3, 3, 2, 2 22.05 kHz 7, 7, 3, 319, 14, 4, 3 24 kHz5, 3, 2, 2, 2, 2, 210, 5, 3, 3, 3, 2, 2 48 kHz5, 3, 2, 2, 2, 2, 2, 210, 5, 3, 3, 3, 3, 2, 2 for 48 kHz [29]. We evaluate two downstream tasks that em- phasize complementary spectral information: automatic speech recognition (ASR), which primarily depends on low-frequency content, and full-band speech reconstruction (SR), which addi- tionally requires high-frequency detail. For the speech recon- struction task, we adapt the original HiFi-GAN vocoder [30]: the vocoder utilizes the representation produced by the frozen pre-trained model and reconstructs the waveform [31, 32]. To ensure a fair comparison across pre-trained models, the down- stream architectures and hyperparameters are identical for each evaluated model. 4.3. Evaluation on SR and ASR To evaluate MSRHuBERT’s applicability across sampling rates, we compare five pre-training configurations: 1) HuBERT Base: Pre-trained only on the 960-hour LibriSpeech at 16 kHz; 2) Re- sampled 16k HuBERT Base: Pre-trained on the full multi-rate pre-training datasets after resampling all speech to 16 kHz; 3) Resampled 24k HuBERT Base: Pre-trained on the full multi-rate pre-training datasets after resampling all speech to 24 kHz; 4) Resampled 48k HuBERT Base: Pre-trained on the full multi-rate pre-training datasets after resampling all speech to 48 kHz; 5) MSRHuBERT: Pre-trained on the full datasets with- out resampling using our proposed architecture. During down- stream fine-tuning and evaluation, we resample each down- stream dataset to the sampling rate expected by the pre-trained model so that frame-level temporal resolution aligns with the pre-training convention and the resolution mismatch is avoided. To directly quantify the cost of violating this assumption, we ad- ditionally conduct experiments in which the downstream dataset is not resampled. Table 2 presents comprehensive results for ASR and SR downstream tasks using different pre-trained models. From these results we draw four main observations: 1) The perfor- mance of the HuBERT depends systematically on the sampling rate used during pre-training. Models pre-trained at 16 kHz im- plicitly focus on low-frequency semantic cues and consequently achieve stronger ASR performance, whereas models pre-trained at 48 kHz retain high-frequency detail and therefore perform better on full-band speech reconstruction, but the additional high-frequency information interferes with the model’s ability to concentrate on low-frequency semantic structure, degrading ASR. 2) Our MSRHuBERT, by design, directly adapts to differ- ent sampling rates without resampling. As a result, it preserves high-frequency information that benefits reconstruction tasks while still maintaining effective modeling of low-frequency se- mantic content required by ASR. 3) Because the ASR train- ing objective is the sampling-rate invariant transcription text, Table 2: The comprehensive results for ASR and SR downstream tasks using different pre-trained models. Model ASR (WER↓)Full-band SR (STOI↑) 16k Hz 22.05k Hz 24k Hz 48k Hz four sampling rates mixed fine-tuning 16k Hz 22.05k Hz 24k Hz 48k Hz 16 kHz 22.05 kHz 24 kHz48 kHz HuBERT Base6.413.346.845.967.423.357.616.6388.4693.0488.5281.42 Re. 16kHz HuBERT Base6.033.156.595.616.892.956.975.5389.3593.4688.4982.49 - w/o resampling in fine-tuning6.034.4715.6138.187.693.7210.3033.7289.3591.8486.7175.53 Re. 24kHz HuBERT Base6.153.286.545.706.953.027.245.7489.6194.3289.3483.75 Re. 48kHz HuBERT Base6.373.416.725.957.283.297.456.1189.7594.0889.0785.93 MSRHuBERT5.893.356.355.836.542.906.825.5690.2694.3889.2585.79 0123456789101112 Layers HuBERT + 16k MSRHuBERT + 16k HuBERT + 48k MSRHuBERT + 48k HuBERT + Mix MSRHuBERT + Mix Settings 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Figure 3: Weight analysis on the ASR task of the SUPERB Benchmark. Layer 0 corresponds to the input of the first Trans- former layer. The y-axis represents different settings, including pre-trained model and sampling rate of the downstream dataset in ASR. we also conduct a mixed-rate fine-tuning experiment, and our method exhibits promising performance. This empirical result supports the feasibility of mixed-rate training for tasks whose labels are independent of sampling rate. 4) When the resolution mismatch exists between the sampling rate used for pre-training and the rate used for fine-tuning, the convention learned during pre-training is violated and performance degrades. In addition, degradation increases with the magnitude of the sampling rate discrepancy. 4.4. Exploration for Learning Paradigm Architecturally, MSRHuBERT retains HuBERT’s core learning paradigm: mask prediction objective and the Transformer en- coder remain unchanged. Based on the essence of the prob- lem, we address the resolution mismatch problem by replac- ing the fixed single-rate CNN with a multi-sampling-rate adap- tive downsampling CNN that operates directly on time-domain waveforms. As a result, existing layer-wise analyses [33] and improvements developed for HuBERT can be transferred to MSRHuBERT with minimal modification. We verify this view- point both qualitatively and quantitatively. Following the SUPERB policies, we apply a weighted sum to the hidden states of different layers and feed it to the task- specific layers. Fig. 3 shows the weights of different layers of HuBERT and MSRHuBERT models on the ASR task. The larger weight indicates the greater contribution of the corre- Table 3: The result of MSRHuBERT with adaptation of inter- mediate layer supervision and progressive decoupling in down- stream fine-tuning ASR task. Model ASR (WER↓) 16k Hz 22.05k Hz 24k Hz 48k Hz Re. 16kHz HuBERT Base6.033.156.595.61 MSRHuBERT5.893.356.355.83 + Intermediate Layer Supervision 5.563.176.165.67 + Progressive Decoupling 5.633.305.945.52 sponding layer. The two models exhibit qualitatively similar weight distributions, with dominant contributions arising from the same middle-to-upper layer region [12]. This correspon- dence indicates that MSRHuBERT sustains the layer-wise rep- resentational structure that HuBERT relies on. To validate compatibility with existing methods for Hu- BERT, we introduce two representative enhancements, interme- diate layer supervision [34] and progressive decoupling [35], developed for HuBERT to MSRHuBERT and evaluate their ef- fects. Table 3 reports the fine-tuning ASR results: both im- provements yield additional performance gains when applied to MSRHuBERT. These quantitative results confirm that MSRHu- BERT retains HuBERT’s training paradigm and therefore bene- fits from prior advances, demonstrating practical transferability and generalization across sampling rates. Moreover, our ap- proach can be easily adapted to other sampling-rate scenarios, such as 32 kHz or 44.1 kHz, with minimal modifications to the multi-sampling-rate adaptive downsampling CNN architecture. Specifically, we choose a downsampling factor matched to each input sampling rate so that all waveforms are compressed to a common temporal resolution prior to subsequent processing. 5. Conclusion This paper proposes the resolution mismatch problem for speech SSL across diverse sampling rates and presents MSRHu- BERT to mitigate this issue, using a self-supervised pre- training framework that introduces a multi-sampling-rate adap- tive downsampling CNN without resampling and holds the core learning paradigm. A series of experiments have been carried out to show that our method preserves high-frequency detail and maintains effective modeling of low-frequency semantic con- tent, sustaining the core learning paradigm and the layer-wise representational structure. 6. References [1] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre- training of deep bidirectional transformers for language under- standing,” in CNACACL: human language technologies, volume 1 (long and short papers), 2019, p. 4171–4186. [2] L. Jing and Y. Tian, “Self-supervised visual feature learning with deep neural networks: A survey,” IEEE TPAMI, vol. 43, no. 11, p. 4037–4058, 2020. [3] A. Mohamed, H.-y. Lee, L. Borgholt, J. D. Havtorn, J. Edin, C. Igel, K. Kirchhoff, S.-W. Li, K. Livescu, L. Maaløe et al., “Self-supervised speech representation learning: A review,” IEEE JSTSP, vol. 16, no. 6, p. 1179–1210, 2022. [4] T. Wang, X. Chen, Z. Chen, S. Yu, and W. Zhu, “An adapter based multi-label pre-training for speech separation and enhancement,” in ICASSP. IEEE, 2023, p. 1–5. [5] J. Lin, M. Ge, W. Wang, H. Li, and M. Feng, “Selective hubert: Self-supervised pre-training for target speaker in clean and mix- ture speech,” IEEE Signal Processing Letters, vol. 31, p. 1014– 1018, 2024. [6] J. Shi, H. Inaguma, X. Ma, I. Kulikov, and A. Sun, “Multi- resolution hubert:Multi-resolution speech self-supervised learningwithmaskedunitprediction,”arXivpreprint arXiv:2310.02720, 2023. [7] J. Zhao and W.-Q. Zhang, “Improving automatic speech recogni- tion performance for low-resource languages with self-supervised models,” IEEE JSTSP, vol. 16, no. 6, p. 1227–1241, 2022. [8] Z. Huang, S. Watanabe, S.-w. Yang, P. Garc ́ ıa, and S. Khudan- pur, “Investigating self-supervised learning for speech enhance- ment and separation,” in ICASSP. IEEE, 2022, p. 6837–6841. [9] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in International conference on machine learning. PMLR, 2023, p. 28 492–28 518. [10] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representa- tions,” NIPS, vol. 33, p. 12 449–12 460, 2020. [11] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdi- nov, and A. Mohamed, “Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,” IEEE/ACM TASLP, vol. 29, p. 3451–3460, 2021. [12] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao et al., “Wavlm: Large-scale self- supervised pre-training for full stack speech processing,” IEEE JSTSP, vol. 16, no. 6, p. 1505–1518, 2022. [13] R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, “Common voice: A massively-multilingual speech corpus,” arXiv preprint arXiv:1912.06670, 2019. [14] X. Li, Q. Wang, and X. Liu, “Masksr: Masked language model for full-band speech restoration,” in Interspeech, 2024, p. 2275– 2279. [15] H. Schroter, A. N. Escalante-B, T. Rosenkranz, and A. Maier, “Deepfilternet: A low complexity speech enhancement frame- work for full-band audio based on deep filtering,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, p. 7407–7411. [16] V. Abrol, A. Thakur, A. Gupta, X. Liu, and S. Shah, “Sampling rate adaptive speaker verification from raw waveforms,” in Inter- national Conference on Pattern Recognition. Springer, 2024, p. 367–382. [17] J. Yu and Y. Luo, “Efficient monaural speech enhancement with universal sample rate band-split rnn,” in ICASSP.IEEE, 2023, p. 1–5. [18] J. Paulus and M. Torcoli, “Sampling frequency independent dia- logue separation,” in EUSIPCO. IEEE, 2022, p. 160–164. [19] G. Yang, Z. Ma, Z. Zheng, Y. Song, Z. Niu, and X. Chen, “Fast- hubert: An efficient training framework for self-supervised speech representation learning,” in ASRU. IEEE, 2023, p. 1–7. [20] J. Yu, L. Yang, N. Xu, J. Yang, and T. Huang, “Slimmable neural networks,” in ICLR. [21] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” arXiv preprint arXiv:1607.06450, 2016. [22] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: an asr corpus based on public domain audio books,” in ICASSP. IEEE, 2015, p. 5206–5210. [23] H. Dubey, A. Aazami, V. Gopal, B. Naderi, S. Braun, R. Cutler, A. Ju, M. Zohourian, M. Tang, M. Golestaneh et al., “Icassp 2023 deep noise suppression challenge,” IEEE Open Journal of Signal Processing, vol. 5, p. 725–737, 2024. [24] M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grang- ier, and M. Auli, “fairseq: A fast, extensible toolkit for sequence modeling,” in CNACACL, 2019, p. 48–53. [25] T.-Q. Lin, H.-y. Lee, and H. Tang, “Melhubert: A simplified hu- bert on mel spectrograms,” in ASRU. IEEE, 2023, p. 1–8. [26] S.-w. Yang, P.-H. Chi, Y.-S. Chuang, C.-I. J. Lai, K. Lakho- tia, Y. Y. Lin, A. T. Liu, J. Shi, X. Chang, G.-T. Lin et al., “Superb: Speech processing universal performance benchmark,” arXiv preprint arXiv:2105.01051, 2021. [27] K. Ito and L. Johnson, “The lj speech dataset,” https://keithito. com/LJ-Speech-Dataset/, 2017. [28] H. Zen, V. Dang, R. Clark, Y. Zhang, R. J. Weiss, Y. Jia, Z. Chen, and Y. Wu, “Libritts: A corpus derived from librispeech for text- to-speech,” Interspeech, 2019. [29] C. Veaux, J. Yamagishi, K. MacDonald et al., “Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit,” Uni- versity of Edinburgh. The Centre for Speech Technology Research (CSTR), vol. 6, p. 15, 2017. [30] J. Kong, J. Kim, and J. Bae, “Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,” NIPS, vol. 33, p. 17 022–17 033, 2020. [31] T. Tan, S. Liu, Y. Duan, S. Zhao, and X. Shao, “System descrip- tion: Speaker anonymization system with sentiment transfer and feature interpolation,” voiceprivacychallenge. org, 2024. [32] A. D ́ efossez, J. Copet, G. Synnaeve, and Y. Adi, “High fidelity neural audio compression,” arXiv preprint arXiv:2210.13438, 2022. [33] A. Pasad, B. Shi, and K. Livescu, “Comparative layer-wise anal- ysis of self-supervised speech models,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, p. 1–5. [34] C. Wang, Y. Wu, S. Chen, S. Liu, J. Li, Y. Qian, and Z. Yang, “Improving self-supervised learning for speech recognition with intermediate layer supervision,” in ICASSP.IEEE, 2022, p. 7092–7096. [35] T. Wang, J. Li, Z. Ma, R. Cao, X. Chen, L. Wang, M. Ge, X. Wang, Y. Wang, J. Dang et al., “Progressive residual extraction based pre-training for speech representation learning,” IEEE TASLP, 2025.