Paper deep dive
Whisper-Aware LLM: Self-Supervised Uncertainty Learning for Robust Whispered Speech Recognition
Gaopeng Xu, Zhenyu Wang, Zheng Xue, Yinfeng Xia, Haitao Yao
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/16/2026, 3:28:18 AM
Summary
The paper introduces the Whisper-Aware LLM, a framework that enhances Automatic Speech Recognition (ASR) for whispered speech by integrating an Uncertainty Perception Module (UPM) into an Audio-LLM. The UPM uses self-supervised tasks (F0 contour prediction and masked spectrum reconstruction) to quantify signal uncertainty. This uncertainty information is operationalized via a Confidence-Fused Decoding mechanism, which modulates the LLM decoder's attention and provides global instructions. The model achieves state-of-the-art performance on whispered speech benchmarks (AISHELL6-Whisper, wTIMIT) and significantly reduces hallucination rates on noisy inputs.
Entities (10)
Relation Signals (7)
Whisper-Aware LLM → achievessotaon → AISHELL6-Whisper
confidence 95% · The model sets a new state-of-the-art on whispered speech with a 17% relative CER reduction on AISHELL6-Whisper.
Whisper-Aware LLM → builton → Qwen2-Audio
confidence 95% · Our framework is built upon the Qwen2-Audio model [16].
Whisper-Aware LLM → reducesmetric → Hallucination Rate
confidence 95% · hallucination rates dropping from over 25% to 4.5%.
Whisper-Aware LLM → usescomponent → Uncertainty Perception Module
confidence 95% · Our framework enhances a standard Audio-LLM by integrating a lightweight Uncertainty Perception Module (UPM)
Whisper-Aware LLM → usesmethod → Confidence-Fused Decoding
confidence 95% · This learned uncertainty is then operationalized via a novel Confidence-Fused Decoding mechanism
Uncertainty Perception Module → performstask → Masked Spectrum Reconstruction
confidence 90% · a second head f Spec is trained to reconstruct randomly masked frames in H enc
Uncertainty Perception Module → performstask → F0 Contour Prediction
confidence 90% · we train a prediction head f F0 within the UPM to predict the F0 contour
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The signal ambiguity of whispered speech drives ASR systems toward two opposing failure modes: failing to capture whispered speech or hallucinatory transcription of noise. This paper introduces the Whisper-Aware LLM, a framework that teaches an Audio-LLM to perceive and react to this uncertainty. Our model develops an intrinsic self-awareness by learning to quantify the physical deficiencies of acoustic signals through targeted self-supervised tasks. This learned uncertainty is then operationalized via a novel Confidence-Fused Decoding mechanism, which provides both high-level instructions and frame-level attention modulation to the LLM decoder. Our experiments confirm the effectiveness of this approach. The model sets a new state-of-the-art on whispered speech with a 17% relative CER reduction on AISHELL6-Whisper. At the same time, it directly addresses the reliability trade-off, with hallucination rates dropping from over 25% to 4.5%.
Tags
Links
- Source: https://arxiv.org/abs/2608.10836v1
- Canonical: https://arxiv.org/abs/2608.10836v1
Trouble viewing inline? Open PDF directly →
Full Text
22,711 characters extracted from source content.
Expand or collapse full text
Whisper-Aware LLM: Self-Supervised Uncertainty Learning for Robust Whispered Speech Recognition Gaopeng Xu 1 , Zhenyu Wang 1 , Zheng Xue 1 , Yinfeng Xia 1 , Haitao Yao 1,∗ 1 Qwen Business Unit of Alibaba, China longque.xgp, wzy497957, jeremy.xz, xiayinfeng.xyf, timmy.yht@alibaba-inc.com Abstract The signal ambiguity of whispered speech drives ASR systems toward two opposing failure modes: failing to capture whis- pered speech or hallucinatory transcription of noise. This paper introduces the Whisper-Aware LLM, a framework that teaches an Audio-LLM to perceive and react to this uncertainty. Our model develops an intrinsic self-awareness by learning to quan- tify the physical deficiencies of acoustic signals through tar- geted self-supervised tasks. This learned uncertainty is then operationalized via a novel Confidence-Fused Decoding mech- anism, which provides both high-level instructions and frame- level attention modulation to the LLM decoder. Our experi- ments confirm the effectiveness of this approach. The model sets a new state-of-the-art on whispered speech with a 17% rel- ative CER reduction on AISHELL6-Whisper. At the same time, it directly addresses the reliability trade-off, with hallucination rates dropping from over 25% to 4.5%. Index Terms: Whisper speech recognition, Audio-LLM, Self- Supervised learning, Uncertainty estimation 1. Introduction Whispered speech, a mode of phonation without vocal fold vi- bration, presents a formidable and long-standing challenge for Automatic Speech Recognition (ASR) systems. The acoustic manifestation of whispering characterized by the absence of a fundamental frequency (F0) and harmonic structure, and a noise-like quality [1, 2] creates a fundamental ambiguity that conventional ASR models struggle to resolve. This ambigu- ity gives rise to a debilitating accuracy-reliability trade-off. On one hand, models trained primarily on normal speech exhibit perceptual failure when encountering whispers, leading to high error rates. On the other hand, increasing a model’s sensitivity to capture these faint cues paradoxically heightens its suscepti- bility to hallucination, where it erroneously transcribes speech from ambient noise. Existing research has attempted to mitigate this trade-off via two principal avenues.The first is a data-centric ap- proach focusing on augmentation, such as generating “pseudo- whisper” data [1, 3]. While beneficial, these methods are lim- ited by the inevitable distributional gap between synthetic and authentic whispered speech.The second avenue is model- centric static adaptation, exemplified by the projection layer proposed by Li et al. [2]. While an improvement, such “one- size-fits-all” static solutions fail to dynamically adjust to the vast spectrum of whisper variability. Crucially, these prevailing paradigms primarily address the acoustic symptoms of whispered speech, rather than its underly- ** indicates the corresponding author. ing cause: a fundamental increase in signal uncertainty. A piv- otal acoustic study by Lin et al. [1] experimentally confirmed that the lack of glottal information is the single most critical fac- tor degrading recognition performance. This key finding sug- gests a new direction: instead of forcing the model to make high-confidence predictions from highly uncertain signals, a more robust approach is to first teach the model to quantify the signal’s intrinsic uncertainty itself. This paper is built upon this principle. We introduce the Whisper-Aware LLM, a framework designed to endow a na- tive Audio-LLM with an intrinsic self-awareness of signal un- certainty. We posit that a model can learn to perceive this uncertainty in a self-supervised manner by being trained to identify the very physical deficiencies that define whispered speech. To this end, we design two innovative self-supervised tasks: F0 contour prediction and masked spectrum reconstruc- tion. The quantified uncertainty is then operationalized through our Confidence-Fused Decoding mechanism. To ensure stable learning, we introduce a disciplined three-stage training strat- egy. Our contributions are: 1. A novel framework integrating self-supervised uncertainty perception directly into an Audio-LLM. 2. Two physics-informed self-supervised tasks for label-free uncertainty learning. 3. A Confidence-Fused Decoding mechanism for dynamic gen- erative control. 4. A systematic three-stage training protocol for robust repre- sentation learning. 2. The Whisper-Aware LLM Framework Our framework enhances a standard Audio-LLM by integrat- ing a lightweight Uncertainty Perception Module (UPM) and a corresponding Confidence-Fused Decoding mechanism. 2.1. Overall Architecture As shown in Fig.1, the architecture consists of three main stages: • Acoustic Encoding: The encoder E maps an input au- dio waveform w into a sequence of representations H enc ∈ R T×D model . • Uncertainty Perception: Our UPM,U , takes H enc as input and produces a global uncertainty vector u global and a frame- wise confidence sequence H conf . • Confidence-Fused Decoding: The decoderD generates the transcription Y autoregressively, guided by the UPM’s out- puts. arXiv:2608.10836v1 [cs.SD] 11 Aug 2026 Feed-Forward Self-Attention Input-Processor F0 Counter Prediction Masked Spectrum Reconstruction Acoustic Encoder(E) Uncertainty Perception Module(UPM) Input Audio W Confidence-Fused LLM Decoder(D) Global Uncertainty Vector Frame-wise Confidence Whisper-Aware LLM MLP WI TR [MS] Acoustic Representations [ENC] [FP0] Final Transcription Attention Modulation (Additive Bais) Instruction Embedding [SA] [IP] [FFN] (� 푆푝 ) � 표� (� �0 ) � 푔푙표푙 SR Speech Representation TR SR WI Speech Representation Text Representation Whisper Indicator Adapter Figure 1: The overall architecture of the Whisper-Aware LLM. The model first encodes the audio into acoustic representations. Our UPM then processes these representations to perceive signal quality, producing a global instruction embedding (e instruct ) and a frame- wise confidence sequence (H conf ). These outputs guide the LLM decoder: the instruction acts as a system prompt, and the confidence scores modulate the attention mechanism. 2.2. Uncertainty Perception Module (UPM) The UPM is a small network trained to perceive signal uncer- tainty through two self-supervised auxiliary tasks. 2.2.1. Task 1: F0 Contour Prediction Motivated by the finding that F0 absence is the most critical factor in whisper recognition degradation [1], we train a pre- diction head f F0 within the UPM to predict the F0 contour ˆ F 0 from H enc . The training loss is the Mean Squared Error (MSE) against a ground-truth contour F 0 : L F0 = 1 T T X t=1 ( ˆ F 0 (t)− F 0 (t)) 2 (1) The frame-wise confidence score h conf (t) is derived from the normalized prediction error e F0 (t): h conf (t) = 1− clip e F0 (t) ε max , 0, 1 (2) where ε max is an error clipping hyperparameter. The resulting H conf is then centered to have a zero mean. 2.2.2. Task 2: Masked Spectrum Reconstruction To capture the unstructured, noise-like nature of the whisper spectrum, a second head f Spec is trained to reconstruct randomly masked frames in H enc . The UPM’s final hidden state, averaged over time, forms the global uncertainty vector u global . 2.3. Confidence-Fused Decoding 2.3.1. Global Instruction Embedding The global uncertainty vector u global is transformed by an MLP[4], f instr , into an instruction embedding e instruct . This is prepended to the decoder’s input sequence, acting as a high- level system prompt. 2.3.2. Attention Modulation In the LLM-based architecture, the acoustic representations H enc are treated as a prefix to the token sequence, and the LLM performs causal self-attention over this entire combined sequence. To guide this process, we introduce an attention bi- asing mechanism. For any query q within the LLM, when it computes an attention score with a key k t that corresponds to the t-th acoustic frame from the encoder, we inject our learned confidence as an additive bias. The modified score is: score(q,k t ) = q ⊤ k t √ d k + w· h conf (t)(3) where w is a learnable scalar. This simple yet effective mech- anism allows the model to dynamically pay less attention to acoustic frames deemed unreliable by the UPM, preventing the decoder from being misled by uncertain evidence. 2.4. Three-Stage Training Strategy To stably integrate our Uncertainty Perception Module (UPM) while avoiding gradient conflicts with the large pre-trained model, we employ a disciplined three-stage training strategy: • Stage 1: UPM Pre-training. We first train the UPM exclu- sively on its self-supervised objectives (L F0 + L Spec ), while the entire Audio-LLM backbone remains frozen. This iso- lates the UPM, allowing it to rapidly learn robust uncertainty features from the frozen encoder without destabilizing the main model. • Stage 2: Interface Adaptation. Next, we focus on connect- ing the UPM to the decoder. We jointly train the audio en- coder, the audio-LLM adapter, the UPM, and our new decod- ing interfaces (the MLP f instr and scalar w), while keeping the large LLM decoder frozen. This stage aligns the modalities and teaches the model to use the uncertainty signals. • Stage 3: Full End-to-End Fine-tuning. Finally, the en- tire model is fine-tuned end-to-end using a composite loss (L ASR +λ aux (L F0 +L Spec )). To maintain efficiency, we apply LoRA[5] to the LLM decoder. 3. Experiments To validate the effectiveness of our Whisper-Aware LLM, we conducted a series of comprehensive experiments. We aimed to evaluate its performance on whispered speech recognition, its general ASR capabilities, its robustness against hallucination, and the contribution of its core components. 3.1. Experimental Setup 3.1.1. Datasets and Metrics Training Datasets. Our work leverages two strategically con- structed datasets for different training phases to build a highly robust model. • UPM Pre-training Corpus (for Stage 1): To ensure the UPM learns a comprehensive model of acoustic signals, we constructed a large-scale, mixed-domain corpus. This corpus includes General-Purpose Speech (a 3,000h subset of WenetSpeech[9], a 3,000h subset of GigaSpeech[10], AISHELL-1[11] and LibriSpeech[12]), Whispered Speech (wTIMIT[13], AISHELL6-Whisper[14]), and a 1,000h col- lection of pure noise. • Fine-tuning Set (for Stage 2 & 3): This set consists of the training splits of AISHELL-1, LibriSpeech, wTIMIT, AISHELL6-Whisper, and a 200h subset of the pure noise data to maintain robustness. Evaluation Datasets. We evaluate our model’s perfor- mance across three key dimensions: • Whispered Speech: The official test sets of AISHELL6- Whisper [2] (Chinese) and wTIMIT [15] (English). • General-Purpose ASR: The test sets of AISHELL-1 and Lib- riSpeech to assess general competency. • Reliability and Hallucination: To specifically evaluate hallu- cination under adverse conditions, we created a Noise Hallu- cination Set containing 1000 audio clips. These clips consist of various non-speech sounds (e.g., machine hum, wind) and segments with either no speech or very faint speech heavily masked by noise, designed to be challenging for conventional VADs and prone to trigger false transcriptions. Metrics. The primary performance metrics are Charac- ter Error Rate (CER%) for Chinese and Word Error Rate (WER%)/CER% for English. On the Noise Hallucination Set, we measure the Hallucination Rate (HR%), calculated as the percentage of audio files containing no transcribable speech that produce a non-empty transcription. 3.1.2. Implementation Details • Base Model: Our framework is built upon the Qwen2-Audio model [16]. It consists of an audio encoder, a lightweight projector as the audio-LLM adapter, and a Qwen-7B LLM [17] backbone. • Uncertainty Perception Module (UPM): Our UPM is com- posed of a shared feature extractor (two 1D-CNN layers with GeLU and a single Transformer[18] encoder layer) followed by two task-specific heads for F0 prediction and masked spectrum reconstruction. • Training Details: Our three-stage strategy was implemented using the AdamW[19] optimizer with a learning rate of 1×10 −5 .In Stage 3, the LLM was updated via LoRA (rank=64, alpha=32). The auxiliary loss weight was set to λ aux = 0.1. Stage 1 ran for 100k steps, while Stages 2 and 3 each ran for 20k steps. 3.1.3. Compared Systems We benchmark against internal baselines sharing the Qwen2- Audio backbone and external state-of-the-art (SOTA) systems. The systems S1-S7 are defined as: Whisper-v3 [20] (S1) and its fine-tuned version (S2); a fine-tuned Qwen2-Audio baseline (S3); and other recent SOTA ASR systems (S4-S7). 3.2. Main Results 3.2.1. Whispered Speech Accuracy We first evaluate whispered speech accuracy on both Chinese and English benchmarks to demonstrate the cross-lingual effec- tiveness of our framework. As detailed in Table 2, our model achieves a new state-of- the-art on the challenging AISHELL6-Whisper test set. With a CER of only 1.31% on whispered speech, it significantly sur- passes all compared systems, including the strong Seed-ASR (S7) which scored 1.58%. This represents a relative CER re- duction of 17% over the previous best. Importantly, our model also achieves the lowest error rate on normal speech (0.63%), indicating that its specialization for whispers does not compro- mise its general ASR capabilities. The results on English benchmarks, presented in Table 1, further reinforce these findings. Our model consistently out- performs all baselines across both normal (N) and whispered (W) conditions for US and Singaporean (SG) accents. Notably, while other models show a dramatic increase in WER when transitioning from normal to whispered speech, our model ex- hibits a much more graceful degradation. This demonstrates its enhanced robustness to the acoustic shift caused by whispering, a direct result of its uncertainty-aware architecture. 3.2.2. General ASR Performance To explicitly verify that our model’s specialization for whis- pers does not impair its general capabilities, we summarize its performance on standard, non-whispered benchmarks in Table 3. Our model demonstrates highly competitive performance against state-of-the-art systems. On the AISHELL-1 test set, it achieves a strong CER of 1.34%, placing it on par with other top-tier systems like Funasr (1.22%). On the LibriSpeech-clean set, our model obtains a robust WER of 1.91%, again show- ing performance in the same league as leading models. This confirms that our uncertainty-aware framework successfully en- hances robustness in challenging conditions without sacrificing its proficiency on high-quality, normal speech. Table 1: Detailed performance comparison on English benchmarks (WER%/CER%). Our model demonstrates superior performance on both normal (N) and whispered (W) speech. Model WER (%)CER (%) N US N SG W US W SG N US N SG W US W SG S1: Whisper-v35.258.678.2019.031.032.502.478.85 S2: Whisper-v3 fine-tuned5.587.467.9515.79 0.931.862.377.53 S3: Qwen2-Audio fine-tuned4.956.857.4013.500.881.652.306.20 S4: Aishell6-whisper AVSR4.506.066.3511.63 0.731.331.955.17 S5: Qwen3-ASR[6]4.997.267.0116.460.981.852.457.11 S6: Funasr-ASR[7]4.656.968.4819.29 0.911.762.888.24 S7: Seed-ASR[8]4.296.516.8812.950.851.622.156.03 Ours4.155.895.9210.810.701.251.854.98 Table 2: Performance on the AISHELL6-Whisper set (CER%). Model CER (%) NormalWhisper S1: Whisper-v33.9518.93 S2: Whisper-v3 fine-tuned1.626.69 S3: Qwen2-Audio fine-tuned1.053.98 S4: Aishell6-whisper AVSR1.114.13 S5: Qwen3-ASR0.643.79 S6: Funasr-ASR0.7219.50 S7: Seed-ASR0.651.58 Ours0.631.31 Table 3: General ASR performance on standard benchmarks. ModelAISHELL-1LibriSpeech-clean (CER%)(WER%) Whisper-v34.721.86 Seed-ASR1.632.80 Funasr1.221.51 Ours1.341.91 3.2.3. Reliability and Hallucination A core contribution of our work is resolving the accuracy- reliability trade-off. We test this on our curated Noise Hallu- cination Set, where an ideal system should produce no output. As shown in Table 4, standard fine-tuned models exhibit a high Hallucination Rate (HR). For instance, even strong baselines like S7 and S5 incorrectly generate text for over 25% of the files, with the highest rate reaching 29.3% (S6). This highlights their tendency to “imagine” speech when faced with ambiguous signals. In stark contrast, our Whisper-Aware LLM achieves an HR of only 4.5%, a substantial reduction. This remarkable improvement demonstrates that by learning to perceive signal uncertainty, our model effectively learns when not to speak, drastically mitigating the risk of generating spurious text from non-speech audio. This reduction is primarily attributed to the Global Instruction mechanism, which guides the decoder, as quantified in our ablation study. Table 4: Reliability evaluation on the Noise Hallucination Set. ModelHR(%) S3: Qwen2-Audio fine-tuned35.5 S5: Qwen3-ASR25.2 S6: Funasr-ASR29.3 S7: Seed-ASR25.2 Ours4.5 3.3. Ablation of Model Components To dissect the contribution of our proposed components, we conducted an ablation study on the AISHELL6-Whisper, with results in Table 5.Starting from the S3 baseline (3.98% CER), using only the frame-wise Attention Modulation yields a clear improvement to 3.45%. The Global Instruction, however, proves to be the primary driver of performance, slashing the CER to 1.84% by providing high-level, strategic awareness of signal quality. Finally, the Full Model combines both, achieving the best performance at 1.31%. Table 5: Ablation of Confidence-Fused Decoding components on AISHELL6-Whisper (Whisper subset). ConfigurationCER(%) Baseline (Vanilla Finetuning)3.98 + Attention Modulation Only3.45 + Global Instruction Only1.84 + Full Model (Ours)1.31 4. Conclusion In this paper, we introduced the Whisper-Aware LLM, a frame- work designed to handle the uncertainty inherent in whispered speech. Instead of just matching patterns, our approach teaches the model to first perceive signal quality through targeted self- supervised tasks. This learned awareness is then used by our Confidence-Fused Decoding mechanism to guide the LLM, al- lowing it to selectively trust the acoustic evidence it receives. Our experiments confirm the success of this strategy. The model achieves state-of-the-art results on whispered speech bench- marks and drastically reduces hallucinations, all while remain- ing a highly effective general-purpose ASR system. 5. Generative AI Use Disclosure The authors used generative AI tools solely for language editing and grammar polishing. All technical content, methodology, experimental design, and analysis were conducted entirely by the authors. 6. References [1] Z. Lin, T. Patel, and O. Scharenborg, “Improving whispered speech recognition performance using pseudo-whispered based data augmentation,” in 2023 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), 2023, p. 1–8. [2] C. Li, F. Su, J. Liu, H. Bu, Y. Wan, H. Suo, and M. Li, “Aishell6-whisper: A chinese mandarin audio-visual whisper speech dataset with speech recognition baselines,” arXiv preprint arXiv:2403.19833, 2024. [3] P. R. Gudepu, G. P. Vadisetti, A. Niranjan, K. Saranu, R. Sarma, M. A. B. Shaik, and P. Paramasivam, “Whisper augmented end- to-end/hybrid speech recognition system—cyclegan approac,” in Proc. Interspeech 2020, 2020, p. 2302–2306. [4] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representations by back-propagating errors,” nature, vol. 323, no. 6088, p. 533–536, 1986. [5] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen et al., “Lora: Low-rank adaptation of large language models.” Iclr, vol. 1, no. 2, p. 3, 2022. [6] X. Shi, X. Wang, Z. Guo, Y. Wang, P. Zhang, X. Zhang, Z. Guo, H. Hao, Y. Xi, B. Yang et al., “Qwen3-asr technical report,” arXiv preprint arXiv:2601.21337, 2025. [7] K. An, Y. Chen, Z. Chen, C. Deng, Z. Du, C. Gao, Z. Gao, B. Gong, X. Li, Y. Li et al., “Fun-asr technical report,” arXiv preprint arXiv:2509.12508, 2025. [8] Y. Bai, J. Chen, J. Chen, W. Chen, Z. Chen, C. Ding, L. Dong, Q. Dong, Y. Du, K. Gao et al., “Seed-asr: Understanding diverse speech and contexts with llm-based speech recognition,” arXiv preprint arXiv:2407.04675, 2024. [9] B. Zhang, H. Lv, P. Guo, Q. Shao, C. Yang, L. Xie, X. Xu, H. Bu, X. Chen, C. Zeng et al., “Wenetspeech: A 10000+ hours multi- domain mandarin corpus for speech recognition,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, p. 6182–6186. [10] G. Chen, S. Chai, G. Wang, J. Du, W.-Q. Zhang, C. Weng, D. Su, D. Povey, J. Trmal, J. Zhang et al., “Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio,” arXiv preprint arXiv:2106.06909, 2021. [11] H. Bu, J. Du, X. Na, B. Wu, and H. Zheng, “Aishell-1: An open- source mandarin speech corpus and a speech recognition base- line,” in 2017 20th conference of the oriental chapter of the inter- national coordinating committee on speech databases and speech I/O systems and assessment (O-COCOSDA).IEEE, 2017, p. 1–5. [12] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: an asr corpus based on public domain audio books,” in 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015, p. 5206–5210. [13] B. P. Lim, Computational differences between whispered and non- whispered speech.University of Illinois at Urbana-Champaign, 2011. [14] C. Li, F. Su, J. Liu, H. Bu, Y. Wan, H. Suo, and M. Li, “Aishell6-whisper: A chinese mandarin audio-visual whisper speech dataset with speech recognition baselines,” arXiv preprint arXiv:2509.23833, 2025. [15] B. P. Lim, “Computational differences between whispered and non-whispered speech,” Ph.D. dissertation, University of Illinois at Urbana-Champaign, 2011. [16] Y. Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y. Leng, Y. Lv, J. He, J. Lin et al., “Qwen2-audio technical report,” arXiv preprint arXiv:2407.10759, 2024. [17] J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang et al., “Qwen technical report,” arXiv preprint arXiv:2309.16609, 2023. [18] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017. [19] I. Loshchilov and F. Hutter, “Decoupled weight decay regulariza- tion,” arXiv preprint arXiv:1711.05101, 2017. [20] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in International conference on machine learning. PMLR, 2023, p. 28 492–28 518.