Paper deep dive
Whisper-MLA: Reducing GPU Memory Consumption of ASR Models based on MHA2MLA Conversion
Sen Zhang, Jianguo Wei, Wenhuan Lu, Xianghu Yue, Wei Li, Qiang Li, Pengcheng Zhao, Ming Cai, Luo Si
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/20/2026, 5:19:16 AM
Summary
The paper introduces Whisper-MLA, a novel architecture that adapts Multi-Head Latent Attention (MLA) to the Whisper Automatic Speech Recognition (ASR) model to reduce GPU memory consumption. By converting the standard Multi-Head Attention (MHA) to MLA, specifically targeting the decoder's self-attention module, the method reduces Key-Value (KV) cache size by up to 87.5% while maintaining competitive accuracy on the LibriSpeech benchmark. The approach utilizes a parameter-efficient conversion strategy with minimal fine-tuning, addressing the memory bottlenecks associated with long-form audio processing.
Entities (10)
Relation Signals (8)
Whisper-MLA → evaluatedon → LibriSpeech
confidence 98% · Extensive experiments on the LibriSpeech benchmark validate the effectiveness of this conversion
Whisper-MLA → isbasedon → Whisper
confidence 98% · We introduce Whisper-MLA, a novel architecture that incorporates Multi-Head Latent Attention (MLA) into the Whisper model.
Whisper → hasmechanism → Multi-Head Attention
confidence 95% · its Multi-Head Attention (MHA) mechanism results in significant GPU memory consumption
Whisper-MLA → reduces → KV cache
confidence 95% · demonstrating that Whisper-MLA reduces the KV cache size by up to 87.5%
Whisper-MLA → usesmechanism → Multi-head Latent Attention
confidence 95% · incorporates Multi-Head Latent Attention (MLA) into the Whisper model
Multi-head Latent Attention → reduces → GPU Memory Consumption
confidence 92% · MLA significantly reduces the KV cache during inference
Whisper-MLA → outperforms → Whisper
confidence 90% · Whisper-MLA consistently uses less memory... Whisper’s memory demand exceeds the 24GB GPU limit... whereas Whisper-MLA operates comfortably
MHA2MLA → inspiredby → DeepSeek
confidence 85% · To facilitate the adoption of MLA, the MHA2MLA framework was proposed... DeepSeek introduced Multi-Head Latent Attention
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The Transformer-based Whisper model has achieved state-of-the-art performance in Automatic Speech Recognition (ASR). However, its Multi-Head Attention (MHA) mechanism results in significant GPU memory consumption due to the linearly growing Key-Value (KV) cache usage, which is problematic for many applications especially with long-form audio. To address this, we introduce Whisper-MLA, a novel architecture that incorporates Multi-Head Latent Attention (MLA) into the Whisper model. Specifically, we adapt MLA for Whisper's absolute positional embeddings and systematically investigate its application across encoder self-attention, decoder self-attention, and cross-attention modules. Empirical results indicate that applying MLA exclusively to decoder self-attention yields the desired balance between performance and memory efficiency. Our proposed approach allows conversion of a pretrained Whisper model to Whisper-MLA with minimal fine-tuning. Extensive experiments on the LibriSpeech benchmark validate the effectiveness of this conversion, demonstrating that Whisper-MLA reduces the KV cache size by up to 87.5% while maintaining competitive accuracy.
Tags
Links
- Source: https://arxiv.org/abs/2603.00563v1
- Canonical: https://arxiv.org/abs/2603.00563v1
Trouble viewing inline? Open PDF directly →
Full Text
19,620 characters extracted from source content.
Expand or collapse full text
WHISPER-MLA: REDUCING GPU MEMORY CONSUMPTION OF ASR MODELS BASED ON MHA2MLA CONVERSION Sen Zhang 1 , Jianguo Wei 1 , Wenhuan Lu 1 , Xianghu Yue 1,† , Wei Li 2 , Qiang Li 2 , Pengcheng Zhao 2 , Ming Cai 2 , Luo Si 2 1 College of Intelligence and Computing, Tianjin University, Tianjin, China 2 Banma Network Technology Co., Ltd. ABSTRACT The Transformer-based Whisper model has achieved state- of-the-art performance in Automatic Speech Recognition (ASR). However, its Multi-Head Attention (MHA) mecha- nism results in significant GPU memory consumption due to the linearly growing Key-Value (KV) cache usage, which is problematic for many applications especially with long-form audio. To address this, we introduce Whisper-MLA, a novel architecture that incorporates Multi-Head Latent Attention (MLA) into the Whisper model. Specifically, we adapt MLA for Whisper’s absolute positional embeddings and systemati- cally investigate its application across encoder self-attention, decoder self-attention, and cross-attention modules.Em- pirical results indicate that applying MLA exclusively to decoder self-attention yields the desired balance between per- formance and memory efficiency. Our proposed approach allows conversion of a pretrained Whisper model to Whisper- MLA with minimal fine-tuning. Extensive experiments on the LibriSpeech benchmark validate the effectiveness of this conversion, demonstrating that Whisper-MLA reduces the KV cache size by up to 87.5% while maintaining competitive accuracy. Index Terms— Automatic Speech Recognition, Whisper, Multi-Head Latent Attention, GPU memory, KV cache 1. INTRODUCTION The Transformer architecture, anchored by its Multi-Head At- tention (MHA) mechanism, has driven substantial advances in Automatic Speech Recognition (ASR) systems [1, 2, 3]. OpenAI’s Whisper model [4], for instance, has achieved state- of-the-art accuracy across diverse acoustic conditions. How- ever, this powerful architecture incurs significant GPU mem- ory consumption, a limitation that becomes particularly pro- nounced when processing long-form audio input [5]. A key bottleneck for long-context tasks is the Key-Value (KV) cache inherent in the MHA mechanism [6], which † Corresponding author. Our source code is publicly available at https://github.com/s- sen/WhisperMLA. scales linearly with both sequence length and model size [7]. To address this, DeepSeek [8] introduced Multi-Head Latent Attention (MLA), an attention mechanism that employs low- rank key-value joint compression. MLA significantly reduces the KV cache during inference and has been shown to outper- form standard MHA on various text-based benchmarks. To facilitate the adoption of MLA, the MHA2MLA framework [9] was proposed to convert mainstream MHA- based Large Language Models (LLMs) to the MLA archi- tecture without requiring full pretraining from scratch. By leveraging techniques like partial rotary position embedding (partial RoPE) and low-rank approximation, MHA2MLA maximizes parameter reuse from pretrained models, thereby reducing inference costs while preserving performance. In the domain of speech processing, the memory chal- lenge is further exacerbated, making efficient attention mech- anisms particularly critical [10]. To the best of our knowl- edge, while MLA has proven effective for textual modalities, its application to encoder-decoder ASR architectures remains largely unexplored. This motivates our work to adapt MLA for ASR, specifically targeting the Whisper model to reduce its inference memory footprint. However, the MHA2MLA framework is incompatible with Whisper due to fundamental architectural differences. The standard MLA formulation is predicated on relative positional encodings [11, 12], such as Rotary Positional En- coding (RoPE) [13], where keys and queries are decoupled into ”content” and ”position” vectors to explicitly inject po- sitional information. Directly applying this architecture to Whisper, which employs absolute positional encodings with pre-integrated positional features, would break the consis- tency of the attention mechanism. To address this challenge, we propose a novel MLA design tailored for Whisper. Our approach preserves the model’s original parameters and capabilities to the greatest extent by retaining its absolute positional encoding schemes. Moreover, while MLA has been applied only to self-attention in decoder-only models, we explore its effectiveness within encoder-decoder framework. In particular, we systematically investigate the integration of MLA into encoder self-attention, arXiv:2603.00563v1 [cs.SD] 28 Feb 2026 Fig. 1: Three attention architectures: (a) Original MHA in Whisper (left), (b) Full low-rank compression MLA (middle), (c) Dimension-preserving MLA (right). decoder self-attention, and decoder cross-attention modules, respectively. Our main contributions are summarized as follows: • We propose a novel MLA architecture for models with absolute positional encoding, enabling substantial re- ductions in memory consumption during inference while preserving recognition performance. • We conduct a systematic study of MLA deployment across different attention modules in Whisper and present Whisper-MLA, a variant that achieves an effec- tive balance between memory efficiency and accuracy. • We develop a parameter-efficient conversion strategy that adapts a pre-trained Whisper model to Whisper- MLA with minimal fine-tuning, eliminating the need for costly training from scratch. 2. METHODOLOGY 2.1. MLA structure adapted for Whisper Since query projections do not contribute to the KV cache size during autoregressive inference, we retain Whisper’s original query processing scheme. For keys and values, we introduce two MLA-based processing approaches, as depicted in Fig- ure 1. Analysis in section 4 reveals that while compressing all key and value dimensions into a low-rank latent space is pos- sible (Figure 2b), preserving a small subset of original key di- mensions from compression significantly improves ASR per- formance (Figure 2c). This dimension-preserving design also aligns more closely with the original MLA concept. The Whisper encoder employs sinusoidal positional em- beddings, where consecutive dimension pairs form a fre- quency subspace, encoding position with a sine and cosine of a specific frequency [6]. Therefore, our dimension preser- vation strategies operate on these subspaces rather than indi- vidual dimensions. Given a requirement to retain r frequency subspaces, we employ two selection strategies: Uniform sampling: This approach balances high- and low-frequency components by selecting r subspaces with the same geometrically spaced intervals: D uniform =⌊k d h 2r ⌋| 0≤ k < r(1) Head-wise 2-norm contribution: This strategy ranks subspaces based on the insight that the product of the 2- norms of the query and the key vectors serves as an upper bound on their contribution to the attention score [14]. Fol- lowing this insight, we quantify the utility of each frequency subspace by computing the average 2-norm product over multiple training samples. The r subspaces with the highest scores are selected: D 2-norm = top-r 0≤k< d 2 (∥q [2k,2k+1] ∗ ∥k [2k,2k+1] ∗ ∥)(2) We compare full compression and both dimension-preserving strategies in section 4. 2.2. The overall architecture of Whisper-MLA We propose and evaluate two Whisper-MLA architectures: Whisper-MLA (Full): We convert all attention mecha- nisms in the Whisper model (including encoder self-attention, Fig. 2: The method of converting Whisper to Whisper-MLA. decoder self-attention, and cross-attention) to the MLA struc- ture. This configuration tests the general applicability of MLA across different types of attention. Whisper-MLA (Decoder Self-attention Only, DSO): This configuration applies MLA exclusively to the decoder’s self-attention, a targeted approach designed to balance mem- ory efficiency with ASR performance. First, the primary bottleneck during inference is the dynamically growing KV cache, an issue unique to the decoder’s self-attention. The encoder’s KV cache, in contrast, is static after a single forward pass. Consequently, modifying only the decoder self-attention achieves the same KV cache reduction as a full model conversion. Second, this configuration preserves ASR performance by leaving the Whisper encoder unaltered. Given its pre-training on a massive dataset, the encoder’s pa- rameters are highly optimized for extracting robust acoustic representations and are widely adopted in various models [15, 16, 17]. Altering its self-attention mechanism would risk degrading this critical capability. By leaving the encoder and cross-attention untouched, our DSO approach mitigates the memory issue without disrupting critical acoustic modeling. 2.3. Conversion from Whisper to Whisper-MLA Following the methodology of MHA2MLA [9], we pro- pose a parameter-efficient fine-tuning framework that adapts pretrained Whisper to Whisper-MLA without training from scratch. As illustrated in Figure 2, we leave the query projec- tion W q unchanged. We split the key projection W k into a preserved part W kp and a compressible part W kc based on the dimension selection strategies from subsection 2.1. We then perform a joint Singular Value Decomposition (SVD) on the concatenated matrix [W kc , W v ] to obtain a low-rank approximation. The joint SVD factorizes the concatenated matrix as: [W kc , W v ] = U kv Σ kv V ⊤ kv (3) where U kv , V kv ∈R d h ×d kv , Σ kv ∈R d kv ×d kv , the latent projection is then defined as: W uk = Σ 1/2 kv V kv [:, d v :], W uv = Σ 1/2 kv V kv [:, : d v ](4) This approach jointly optimizes a shared latent space for the compressible key components and the values, maximizing parameter reuse from the pretrained Whisper model. 3. EXPERIMENTAL SETUP We employ the official Whisper-small model as our baseline, which contains 244 million parameters and was pre-trained on 680,000 hours of speech data [4]. Leveraging DeepSeek’s empirical heuristics [8], our dimension-preserving strategy maintains 48 of the 768 key dimensions and applies low-rank approximation to project the remaining key and all value di- mensions into a unified latent space of size 96. We convert the pretrained Whisper model to Whisper- MLA and then fine-tune it on the 960-hour LibriSpeech dataset [18] for 3 epochs. We evaluate our model on the LibriSpeech validation and test sets. All experiments were conducted on a single NVIDIA RTX 4090 GPU (24GB) with a batch size of 8 and a gradient accumulation factor of 4. Under this configuration, convert- ing Whisper to Whisper-MLA requires only 12 hours, high- lighting the practical advantage of our method in resource- constrained scenarios. 4. RESULTS AND ANALYSIS As shown in Table 1, the Whisper-MLA models, generated via our proposed conversion and fine-tuning framework, suc- cessfully preserve the strong recognition capabilities of the original model. For a fair comparison, we also fine-tuned the standard Whisper model on the same dataset. Our results show that Whisper-MLA achieves a substantial KV cache re- duction (up to 87.5%) with minimal impact on ASR perfor- mance. Notably, the best-performing variant, Whisper-MLA (DSO) with uniform sampling, achieves a Word Error Rate (WER) that is highly competitive with the fine-tuned Whisper baseline, with an average WER across the four test sets that is only 0.17% higher. This demonstrates an excellent trade-off between memory efficiency and accuracy. Next, we compare the two architectural variants. Whisper- MLA (Full) shows reasonable ASR capability, confirming that MLA can be applied to encoder self-attention and cross- attention. However, its performance lags significantly behind Whisper-MLA (DSO). We attribute this to the DSO variant retaining more of the original pretrained parameters, espe- cially in the encoder and cross-attention modules, which are Table 1: WER (%) on LibriSpeech and KV Cache Memory Reduction for Whisper and Whisper-MLA Models. ModelKeep StrategyKV Cache Red. dev-clean dev-other test-clean test-other avg Whisper(pretrained)-0%16.3719.7816.0020.9018.36 Whisper(finetuned)-0%6.3214.866.8615.0510.95 Whisper-MLA (Full) Full Compression87.50%16.5827.7916.3728.0922.46 Uniform81.25%12.3219.7213.1821.3416.81 2-Norm81.25%11.7520.8312.1521.7416.82 Whisper-MLA (DSO) Full Compression87.50%8.6916.998.8717.8613.29 Uniform81.25%6.6015.236.6115.3211.12 2-Norm81.25%7.3316.177.8216.1812.06 256512102420484096 1 1.2 1.4 1.6 bsz=1 sequence length GPU Memory Consumption(GB) WhisperWhisper-MLA 256512102420484096 2 3 bsz=4 sequence length 256512102420484096 4 6 8 10 12 bsz=16 sequence length 256512102420484096 15 16 17 bsz=64 oomoomoom sequence length Fig. 3: Comparison of GPU memory consumption between Whisper and Whisper-MLA across different batch sizes and se- quence lengths critical for processing acoustic features. Consequently, we select the DSO architecture as our primary model. We further investigate the impact of different dimension preservation strategies. The results clearly show that preserv- ing even a small subset of key dimensions (6.25% of the total) substantially improves performance over full compression across all settings. For Whisper-MLA (Full), uniform and 2-norm strategies perform comparably. For Whisper-MLA (DSO), the uniform sampling strategy outperforms the 2- norm selection strategy. We infer this is because the Whisper decoder uses learnable positional embeddings, which lack the explicit frequency subspace structure of the encoder’s sinu- soidal embeddings. Thus, we select uniform sampling as the default dimension preservation strategy for Whisper-MLA. Finally, we compare the GPU memory consumption of Whisper and Whisper-MLA under practical inference sce- narios. As shown in Figure 3, Whisper-MLA consistently uses less memory. This gap becomes more pronounced as sequence length and batch size increase. For instance, at a batch size of 16 and a sequence length of 4096, Whisper- MLA consumes only about half of the memory of Whisper. More critically, when the batch size is 64 and the sequence length is 2048, Whisper’s memory demand exceeds the 24GB GPU limit, resulting in out-of-memory (OOM), whereas Whisper-MLA operates comfortably at just 15.4 GB. These results underscore the superiority of Whisper-MLA for long- form speech recognition, especially with large batch sizes in resource-constrained environments. 5. CONCLUSION This paper presents Whisper-MLA, a novel architecture cre- ated through an efficient conversion for the Whisper model, successfully addressing its high memory consumption in ASR. Our approach features a unique adaptation of Multi- Head Latent Attention (MLA) for models with absolute positional embeddings. Through systematic placement ex- periments, we identify that applying MLA solely to the de- coder’s self-attention provides an optimal trade-off. These findings not only validate the effectiveness of our conver- sion methodology for encoder-decoder ASR architectures but also pave the way for deploying large-scale speech models on resource-constrained hardware, particularly for long-form audio applications. 6. ACKNOWLEDGEMENT This work was supported in part by the National Key Re- search and Development Program of China under Grant 2023YFB2603902, in part by the Major Science and Tech- nology Specific Project of Xining under Grant 2024-Z-7. 7. REFERENCES [1] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al., “Conformer: Convolution-augmented transformer for speech recog- nition,” arXiv preprint arXiv:2005.08100, 2020. [2] Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le, “Specaugment: A simple data augmentation method for automatic speech recognition,” arXiv preprint arXiv:1904.08779, 2019. [3] Linhao Dong, Shuang Xu, and Bo Xu,“Speech- transformer:a no-recurrence sequence-to-sequence model for speech recognition,” in 2018 IEEE interna- tional conference on acoustics, speech and signal pro- cessing (ICASSP). IEEE, 2018, p. 5884–5888. [4] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brock- man, Christine McLeavey, and Ilya Sutskever, “Robust speech recognition via large-scale weak supervision,” in International conference on machine learning. PMLR, 2023, p. 28492–28518. [5] Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler, “Efficient transformers: A survey,” ACM Com- put. Surv., vol. 55, no. 6, Dec. 2022. [6] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin, “Attention is all you need,” Ad- vances in neural information processing systems, vol. 30, 2017. [7] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica, “Efficient memory manage- ment for large language model serving with pagedatten- tion,” in Proceedings of the 29th symposium on operat- ing systems principles, 2023, p. 611–626. [8] Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al., “Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model,” arXiv preprint arXiv:2405.04434, 2024. [9] Tao Ji, Bin Guo, Yuanbin Wu, Qipeng Guo, Lixing Shen, Zhan Chen, Xipeng Qiu, Qi Zhang, and Tao Gui, “Towards economical inference: Enabling deepseek’s multi-head latent attention in any transformer-based llms,” arXiv preprint arXiv:2502.14837, 2025. [10] Qian Zhang, Han Lu, Hasim Sak, Anshuman Tripathi, Erik McDermott, Stephen Koo, and Shankar Kumar, “Transformer transducer: A streamable speech recog- nition model with transformer encoders and rnn-t loss,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, p. 7829–7833. [11] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al., “Llama 2: Open foundation and fine-tuned chat mod- els,” arXiv preprint arXiv:2307.09288, 2023. [12] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al., “Qwen technical report,” arXiv preprint arXiv:2309.16609, 2023. [13] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu, “Roformer: Enhanced trans- former with rotary position embedding,” Neurocomput- ing, vol. 568, p. 127063, 2024. [14] FedericoBarbero,AlexVitvitskyi,Christos Perivolaropoulos,RazvanPascanu,andPetar Veli ˇ ckovi ́ c, “Round and round we go! what makes rotary positional encodings useful?,” URL https://arxiv. org/abs/2410.06205, 2025. [15] Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang,“Salmonn: Towards generic hearing abilities for large language models,” arXiv preprint arXiv:2310.13289, 2023. [16] Zhifei Xie and Changqiao Wu,“Mini-omni: Lan- guage models can hear, talk while thinking in stream- ing,” arXiv preprint arXiv:2408.16725, 2024. [17] Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng, “Llama-omni: Seamless speech interaction with large language models,” arXiv preprint arXiv:2409.06666, 2024. [18] Vassil Panayotov, Guoguo Chen, Daniel Povey, and San- jeev Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015, p. 5206–5210.