Paper deep dive
AudioTETRIS: Token Elimination via Transitional Importance Signals for Efficient Audio LLMs
Shreya Sajid, Anna Vettoruzzo, Joaquin Vanschoren
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 10/13/2026, 2:48:50 AM
Summary
The paper introduces AudioTETRIS, a training-free, two-stage framework for compressing audio tokens in Large Audio Language Models (LALMs). It leverages the temporal derivative of audio encoder hidden states to create an acoustic prior that identifies phoneme transitions, fusing this with depth-adaptive attention scores from the LLM decoder. This approach allows for progressive top-k token selection, retaining near-uncompressed performance with only 25% of audio tokens across models like Qwen2-Audio-7B, Phi-4-Multimodal, and Audio Flamingo 3.
Entities (16)
Relation Signals (14)
AudioTETRIS → appliedtotask → Audio Question Answering
confidence 95% · Evaluated on... tasks including... Audio Question Answering...
AudioTETRIS → appliedtotask → Speech Emotion Recognition
confidence 95% · Evaluated on... tasks including... Speech Emotion Recognition...
AudioTETRIS → appliedtotask → Automatic Speech Recognition
confidence 95% · Evaluated on... tasks including Automatic Speech Recognition (ASR)...
AudioTETRIS → appliedtotask → Speech Translation
confidence 95% · Evaluated on... tasks including... Speech Translation.
AudioTETRIS → evaluatedon → Qwen2-Audio-7B
confidence 95% · Evaluated on Qwen2-Audio-7B, Phi-4-Multimodal, and Audio Flamingo 3, AudioTETRIS retains near-uncompressed performance...
AudioTETRIS → evaluatedon → Phi-4-Multimodal
confidence 95% · Evaluated on Qwen2-Audio-7B, Phi-4-Multimodal, and Audio Flamingo 3...
AudioTETRIS → evaluatedon → Audio-Flamingo-3
confidence 95% · Evaluated on Qwen2-Audio-7B, Phi-4-Multimodal, and Audio Flamingo 3...
LibriSpeech → usedfortask → Automatic Speech Recognition
confidence 92% · automatic speech recognition on LibriSpeech test-other
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Audio Language Models (LALMs) achieve strong performance on speech-text tasks, but standard audio encoders produce hundreds of tokens for even short audio clips. Because self-attention scales quadratically in sequence length, this inflates inference-time compute substantially. Current training-free token compression methods prune or merge tokens based solely on coarse attention or feature similarity. However, treating audio tokens as generic, isotropic features smooths over spectral transitions, destroying fine-grained phonetic transitions essential for decoding speech. In contrast, speech signals exhibit highly structured temporal patterns where information density peaks at acoustic transitions. We propose AudioTETRIS, a training-free, two-stage framework that uses the audio encoder's representation geometry as an acoustic prior for token importance before applying attention-based compression. Instead of discarding encoder structure after the encoding stage, AudioTETRIS propagates this prior directly into the LLM decoder to compute a per-token fused score. This score drives progressive top-$k$ token selection across decoder layers without requiring additional training or auxiliary models. Evaluated on Qwen2-Audio-7B, Phi-4-Multimodal, and Audio Flamingo 3, AudioTETRIS retains near-uncompressed performance on several speech tasks using only 25% of audio tokens, matches or surpasses full-token accuracy at moderate compression rates, and consistently outperforms existing training-free baselines.
Tags
Links
- Source: https://arxiv.org/abs/2610.13600v1
- Canonical: https://arxiv.org/abs/2610.13600v1
Trouble viewing inline? Open PDF directly →
Full Text
88,565 characters extracted from source content.
Expand or collapse full text
AudioTETRIS: Token Elimination via Transitional Importance Signals for Efficient Audio LLMs Shreya Sajid Affiliation: Department of Mathematics & Computer Science Affiliation: Eindhoven University of Technology Anna Vettoruzzo & Joaquin Vanschoren Affiliation: Eindhoven, Netherlands Affiliation: s.sajid, a.vettoruzzo, j.vanschoren@tue.nl Abstract Large Audio Language Models (LALMs) achieve strong performance on speech-text tasks, but standard audio encoders produce hundreds of tokens for even short audio clips. Because self-attention scales quadratically in sequence length, this inflates inference-time compute substantially. Current training-free token compression methods prune or merge tokens based solely on coarse attention or feature similarity. However, treating audio tokens as generic, isotropic features smooths over spectral transitions, destroying fine-grained phonetic transitions essential for decoding speech. In contrast, speech signals exhibit highly structured temporal patterns where information density peaks at acoustic transitions. We propose AudioTETRIS, a training-free, two-stage framework that uses the audio encoder’s representation geometry as an acoustic prior for token importance before applying attention-based compression. Instead of discarding encoder structure after the encoding stage, AudioTETRIS propagates this prior directly into the LLM decoder to compute a per-token fused score. This score drives progressive top-k token selection across decoder layers without requiring additional training or auxiliary models. Evaluated on Qwen2-Audio-7B, Phi-4-Multimodal, and Audio Flamingo 3, AudioTETRIS retains near-uncompressed performance on several speech tasks using only 25% of audio tokens, matches or surpasses full-token accuracy at moderate compression rates, and consistently outperforms existing training-free baselines. 1 Introduction Figure 1: Overview of AudioTETRIS. A phoneme-transition-aware acoustic prior wtw_t is computed from the audio encoder’s hidden states via the transitional distance dt=‖ht−ht−1‖22d_t=\|h_t-h_t-1\|_2^2, preserving phoneme transitions yhardy_hard (§4; left). Token pruning occurs in two stages: Stage 1 injects wtw_t as a static acoustic prior from the encoder’s hidden states; Stage 2 computes a depth-adaptive LLM attention score, fusing intra-audio attention in early layers with cross-modal (text-to-audio) attention in later layers (right). The two scores are fused into a single importance score, and top-k selection retains the most salient tokens at each pruning layer ℓ . Large Audio Language Models (LALMs) have achieved state-of-the-art results across several speech-text tasks like speech recognition, audio QA, and general audio understanding (Chu et al., 2024; Radford et al., 2023; Chu et al., 2023; Wang et al., 2025a; Yang et al., 2024). Typically, these models use a pretrained audio encoder to map the input waveform into a sequence of continuous frame-level embeddings (audio tokens), which are projected into the Large Language Model (LLM) embedding space and prepended to text prompts. However, this approach creates a severe computational bottleneck: standard speech encoders generate hundreds of tokens for just a few seconds of audio (Lin et al., 2025; Chu et al., 2024; Ghosh et al., 2025a). Because decoder self-attention scales quadratically with sequence length, this large token footprint drives up inference costs (Tay et al., 2022). Furthermore, uniform audio tokenization is inherently redundant (Radford et al., 2023; Chu et al., 2024; Ghosh et al., 2025a; Wang et al., 2025b; Zhang et al., 2026). It clashes with the nature of speech, where information is concentrated during rapid phonetic transitions rather than steady-state regions (Furui, 1986a; Stevens, 2002; Aylett and Turk, 2004; Zhang et al., 2025a). Reducing token count without losing these acoustically rich transition regions is vital for efficient LALM deployment. While recent methods apply token pruning or merging (Lin et al., 2025; Bhati et al., 2025; Gong et al., 2026; Behera et al., 2024), they share a substantial flaw: they rely solely on isolated decoder attention or local feature similarity, ignoring the structural phoneme transitions that could be already captured by the encoder (Behera et al., 2024; Lin et al., 2025; Gong et al., 2026; Jung et al., 2026; Bhati et al., 2025). We hypothesize that speech token compression should be guided by the underlying acoustic structure of the signal, rather than post-hoc decoder attention alone. Our approach is based on a simple insight: regions of rapid spectral transition in the encoder’s hidden states coincide with phonetic transitions, where essential acoustic details reside (Furui, 1986a; Stevens, 2002). We empirically validate this hypothesis in Section 4, showing a strong correlation between representation variance in encoder space and actual phonetic segment transitions. Compression methods that ignore this structure could risk discarding high-frequency acoustic details and degrading downstream performance (Zhang et al., 2026; Libera et al., 2026). To solve this, we introduce AudioTETRIS (Figure 1), a training-free framework that compresses audio tokens by jointly exploiting speech-specific acoustic structures and decoder-side attention. First, AudioTETRIS derives a phoneme-transition-aware prior from the temporal derivative of hidden states, isolating regions of high spectral change to protect information-dense transition segments. Second, this prior is propagated into the LLM decoder and fused with a depth-adaptive attention score derived from intra-audio attention in early layers and cross-modal text-to-audio attention in middle layers. By coupling acoustic salience with task-conditioned attention, token survival is driven by both structural and semantic relevance. We evaluate AudioTETRIS across backbones like Qwen2-Audio-7B, Phi-4-Multimodal, and Audio Flamingo 3 (Chu et al., 2024; Microsoft et al., 2025; Ghosh et al., 2025a) on tasks including Automatic Speech Recognition (ASR), Speech Emotion Recognition, Audio Question Answering, and Speech Translation. AudioTETRIS retains near-lossless accuracy while keeping as few as 25% of tokens, outperforming other training-free baselines, and even surpassing full-sequence models at moderate compression rates. Our main contributions are: • Acoustic transition prior: We show that temporal variance in encoder hidden states serves as a reliable proxy for phonetic transitions, capturing high-information regions without fine-tuning or external alignments. • AudioTETRIS framework: We propose a training-free compression framework that propagates this acoustic prior into the LLM decoder, dynamically guiding self-attention to preserve both acoustic and semantic details. • State-of-the-art efficiency: Extensive experiments demonstrate near-lossless performance at aggressive compression levels (25% token retention), outperforming strong training-free baselines across diverse benchmarks. 2 Related Work Large Audio Language Models. LALMs typically couple a pretrained speech encoder, such as Whisper (Radford et al., 2023), with an LLM backbone (Chu et al., 2024; Ghosh et al., 2025b; Ghosh et al., 2025a; Microsoft et al., 2025; Tang et al., 2024; Deshmukh et al., 2023), projecting audio features into the text embedding space. However, this paradigm yields audio token sequences that are orders of magnitude longer than the accompanying text, creating severe computational overhead during prefill and generation. Reducing this token count without sacrificing the dense lexical content of the underlying signal remains a key challenge (Shao et al., 2026; He et al., 2025). Training-Free Token Compression. Training-free compression methods eliminate redundant audio tokens without modifying model weights. One direction seeks to prune or merge sequence tokens prior to or during LLM processing (Bolya et al., 2023; Behera et al., 2024; Chen et al., 2024; Jung et al., 2026). Across vision-language models (VLMs), methods like SparseVLM (Zhang et al., 2025b) leverage self-attention logits to select text tokens as ‘raters’ that adaptively prune visual tokens across layers (Shang et al., 2025; Yang et al., 2025; Xing et al., 2025, see also). In the speech domain, SpeechPrune (Lin et al., 2025) compresses long-form speech (≥ 30s) tokens using approximated attention scores from the LLM’s initial transformer layer alongside speech-text feature similarity. However, relying on early-layer LLM dynamics forces the model to evaluate query relevance before building a comprehensive contextual representation of the signal. Alternatively, encoder-level methods like Local Temporal Bipartite Merging (LTBM) (Luo et al., 2026) merge nearby tokens under a temporal window constraint (Li et al., 2023), relying purely on local feature similarity rather than task-specific or phonetic cues. Training-Based & Multimodal Token Compression. Rather than relying on training-free pruning, alternative strategies preserve performance by introducing extra trainable parameters or secondary modalities (Kavaki and Mandel, 2025). For instance, encoder-level pooling methods often require low-rank adapters (LoRA) (Hu et al., 2022) to fine-tune the backbone and recover downstream accuracy (Bhati et al., 2025). Although effective, this training dependency eliminates the module’s utility as a plug-and-play solution for unseen backbones or domain shifts. Other frameworks, such as EchoingPixels (Gong et al., 2026), achieve substantial token reduction by exploiting cross-modal alignment across visual and audio streams, but do not extend naturally to audio-language tasks. Across both directions, training-free methods bypass acoustic structure while structure-aware methods require fine-tuning or secondary modalities - a trade-off AudioTETRIS resolves with a training-free, acoustically grounded importance signal (§3). 3 Methodology We present AudioTETRIS, a training-free framework for compressing audio tokens in LALMs. AudioTETRIS operates in two stages: Stage 1 (§3.2) extracts a phoneme-transition-aware acoustic prior directly from the audio encoder’s hidden states, and Stage 2 (§3.3) propagates this prior into the LLM decoder, fusing it with a depth-adaptive attention score to progressively prune audio tokens across decoder layers. Figure 1 illustrates the pipeline. 3.1 Preliminaries Given an input waveform, the audio encoder outputs a sequence of hidden states h1,…,hT∈ℝT×d\h_1,…,h_T\ ^T× d representing T audio tokens. These tokens are projected into the LLM embedding space and concatenated with the query embeddings to yield the initial decoder sequence X=[x1a,…,xTa,x1q,…,xMq]X=[x_1^a,…,x_T^a,x_1^q,…,x_M^q], with a and q referring to audio and query tokens. We index decoder layers as ℓ=0,…,Ldec−1 =0,…,L_dec-1, where LdecL_dec is the number of decoder layers, and denote the layer-ℓ self-attention weight matrix as A(ℓ)∈ℝH×N(ℓ)×N(ℓ)A^( ) ^H× N^( )× N^( ), where H is the number of heads and N(ℓ)N^( ) is the sequence length at layer ℓ (which shrinks as audio tokens are pruned). 3.2 Stage 1: Phoneme Transition Score Motivation. Linguistic information in speech is concentrated near acoustic-phonetic transitions (Furui, 1986b). Because these transitions carry the most acoustic and phonetic information, we hypothesize that tokens spanning them undergo significantly larger representational shifts in latent space than tokens within stable phonetic segments. We leverage this geometric property to construct an acoustic gate that identifies informative tokens without external supervision. Phonetic Transition Score Formulation. For each audio token t, we compute the token-to-token derivative dtd_t as the squared Euclidean distance between consecutive hidden states ht,ht−1∈ℝdh_t,h_t-1 ^d taken from a single intermediate encoder layer (layer selection in §5.1): dt=‖ht−ht−1‖22,d1:=0,d_t=\|h_t-h_t-1\|_2^2, d_1:=0, (1) where the first token is assigned zero derivative in the absence of a preceding token. The squared L2L_2 norm amplifies sharp representational shifts over intra-segment noise at a linear computational cost of (d)O(d) per token. To modulate attention during pruning, we convert dtd_t into a normalized acoustic weight wtw_t: wt=dt1T∑i=1Tdi,w_t= d_t 1T _i=1^Td_i, (2) where T is the total number of audio tokens in the utterance. The denominator enforces unit mean (t[wt]=1E_t[w_t]=1). 3.3 Stage 2: Depth-Adaptive Attention Scoring Figure 2: Distribution of decoder attention mass allocated to audio tokens across model depth in Qwen2-Audio on LibriSpeech. Despite audio tokens comprising 84.98%84.98\% of the input sequence (dashed ring), early layers are dominated by intra-audio attention (76.91%76.91\%), which drops sharply in middle (46.73%46.73\%) and late layers (40.60%40.60\%) as attention shifts to text-conditioned representations. This shift motivates our layer-dependent pooling rule in Eq. 3. Motivation. Attention mass allocation evolves significantly across LLM depth (Figure 2). Early decoder layers are dominated by intra-audio attention (audio tokens attending to audio tokens) to refine structural audio representations. In contrast, middle and late layers reallocate attention toward cross-modal interaction, where text queries attend directly to audio tokens. Consequently, a single, depth-invariant attention metric fails to capture token importance uniformly across the network. Layer-Adaptive Scoring. We address this depth dependence by defining the attention score p~t(ℓ) p_t^( ) for audio token t at layer ℓ over a depth-specific subset of query rows: p~t(ℓ)=1H⋅|a|∑h=1H∑i∈aAh,i,t(ℓ),ℓ∈ℒearly(intra-audio filter)1H⋅|q|∑h=1H∑i∈qAh,i,t(ℓ),ℓ∈ℒlater(cross-modal filter) p_t^( )= cases 1H·|Q_a| _h=1^H _i _aA^( )_h,i,t,& _early (intra-audio filter)\\[9.47217pt] 1H·|Q_q| _h=1^H _i _qA^( )_h,i,t,& _later (cross-modal filter) cases (3) where aQ_a and qQ_q denote the index sets of audio-query and text-query rows, H is the number of attention heads, and ℒearly,ℒlater\L_early,L_later\ partition the decoder layers. Automated Crossover Discovery. Rather than hand-crafting the layer partition ℒearly,ℒlater\L_early,L_later\, we determine the crossover layer ℓ† from backbone attention dynamics on a small calibration set, as the first depth at which mean cross-modal attention mass across(ℓ)a_cross( ) overtakes mean intra-audio attention mass aintra(ℓ)a_intra( ): ℓ†=minℓ:across(ℓ)≥aintra(ℓ), = \ :a_cross( )≥ a_intra( )\, (4) This unsupervised procedure identifies ℓ† in a single calibration pass per backbone; full derivation and per-layer statistics are in Appendix A.2.2. Score Fusion. At each pruning layer ℓ , we fuse the layer-adaptive attention score p~t(ℓ) p_t^( ) with the static acoustic gate wtw_t: st(ℓ)=p~t(ℓ)⋅wt.s_t^( )= p_t^( )· w_t. (5) At each pruning layer ℓ , we rank tokens by st(ℓ)s_t^( ) and retain a top fraction kℓk_ , discarding the rest (see §5.1 for the pruning schedule and how kℓk_ is set per compression rate). 4 Analysis: Validation of Acoustic State Dynamics In Section 3.2, we hypothesized that token-to-token encoder state transitions dtd_t track actual phonetic transitions. Here, we validate this hypothesis both quantitatively against ground-truth forced alignments and label-free via distributional kurtosis. Phoneme Transition Correlation. To evaluate whether encoder distance spikes align with physical phoneme transitions, we define a continuous diagnostic transition prior btb_t. Because audio encoders distribute context across neighboring tokens (Radford et al., 2023), we convolve dtd_t with a Gaussian kernel of width σ=2σ=2 and clip to [0,1][0,1]: bt=clip(∑τdτ⋅(t−τ, 0,σ2), 0, 1).b_t=clip ( _τd_τ·N(t-τ;\,0,\,σ^2),\ 0,\ 1 ). (6) We evaluate btb_t against forced alignments on LibriSpeech (Panayotov et al., 2015; CorentinJ, 2019). Standard alignment tools yield discrete phonetic transitions yhardy_hard. To model continuous speech coarticulation (Furui, 1986b), we apply identical Gaussian smoothing to yhardy_hard, producing a continuous target ysofty_soft. We measure the Pearson correlations rhard=corr(bt,yhard)r_hard=corr(b_t,y_hard) and rsoft=corr(bt,ysoft)r_soft=corr(b_t,y_soft). Figure 8 in Appendix A.2.3 illustrates this end-to-end pipeline. We provide additional dataset validations in Appendix A.2.4. As shown in Figure 3 (top), transition alignment varies predictably across depth, peaking in mid-network layers across all backbones: Qwen2-Audio (layer 16, rsoft=0.68r_soft=0.68) (Chu et al., 2024), Phi-4-Multimodal (layer 14, rsoft=0.59r_soft=0.59) (Microsoft et al., 2025), and Audio Flamingo 3 (layer 15, rsoft=0.62r_soft=0.62) (Ghosh et al., 2025a). Appendix A.2.5 provides an ablation over σ. Figure 3: Layer-wise diagnostics across Qwen2-Audio, Phi-4-Multimodal, and Audio Flamingo 3. Top: Mean correlation between btb_t and ground-truth phoneme transitions (rsoftr_soft and rhardr_hard). Bottom: Kurtosis (κ) of the distance distribution dtd_t. Both metrics identify an intrinsic mid-network sweet spot (shaded) where phonetic transitions are maximally accentuated. Label-Free Diagnostic via Kurtosis. To verify this mid-network correlation peak without relying on external alignments, we measure the distributional kurtosis (κ) of dtd_t across layer depth. High kurtosis indicates that transitions stand out as sharp, heavy-tailed outliers above background intra-segment noise. As shown in Figure 3 (bottom), kurtosis peaks at the exact same mid-network layers (κ≈24.0κ≈ 24.0-26.526.5), providing an unsupervised method to verify transition isolation. Trajectory Visualization. A qualitative t-SNE analysis of per-token hidden states (Appendix A.2.6, Figure 11) corroborates this pattern: mid-network layers show tight phoneme clustering separated by sudden trajectory jumps at transitions, while early and late layers show progressively less phonetically consolidated structure. 5 Experiments & Results 5.1 Experimental Setup Backbones and Datasets. We evaluate AudioTETRIS on three LALM backbones with distinct encoder-decoder architectures: Qwen2-Audio-7B (∼ 8.4B) (Chu et al., 2024), Phi-4-Multimodal (5.6B) (Microsoft et al., 2025), and Audio Flamingo 3 (∼ 8.3B) (Ghosh et al., 2025a). Our evaluation spans four core tasks: automatic speech recognition on LibriSpeech test-other (Panayotov et al., 2015), emotion recognition on IEMOCAP (Busso et al., 2008), audio question answering on MMAU-mini (Sakshi et al., 2025), and speech translation (English-to-Chinese) on CoVoST 2 (Wang et al., 2021). Together, these benchmarks span lexical transcription, cross-lingual generation, paralinguistic classification, and complex audio reasoning, allowing us to test whether preserving phoneme-transition regions specifically benefits tasks that require precise lexical recovery. Baselines. We compare against SpeechPrune (Lin et al., 2025), SparseVLM (Zhang et al., 2025b), and LTBM (Luo et al., 2026), representing the two families of prior work discussed in §2. We additionally include a Random Pruning baseline that retains a uniformly random token subset at matched compression rates, providing a lower-bound sanity check. All baselines are implemented under matched compression schedules using released hyperparameters where available. Implementation Details. We evaluate four target token compression rates - 0.36,0.51,0.640.36,0.51,0.64, and 0.750.75 - representing the fraction of audio tokens removed by the final decoder layer. These correspond to uniform per-stage retention factors r∈0.8,0.7,0.6,0.5r∈\0.8,0.7,0.6,0.5\ across two pruning layers (1−r21-r^2) in our experiments, ranging from light reduction to aggressive 4×4× compression. Pruning is scheduled at layer 111 1 Decoder layers are indexed starting from 0 throughout this paper (matching the axis convention used in Figures 2 and 7); e.g., ‘layer 1’ denotes the second decoder layer. and at a backbone-specific later layer ℓ† (Eq. 4) via progressive top-k retention on the fused score st(ℓ)s_t^( ) (Eq. 5). The acoustic prior wtw_t is computed from a single mid-network encoder layer per backbone: layer 16 for Qwen2-Audio, layer 14 for Phi-4-Multimodal, and layer 15 for Audio Flamingo 3. These layers are selected by the label-free kurtosis criterion of §4 (the depth maximizing κ of dtd_t); forced alignments are used only to validate this choice post hoc and are not required to apply AudioTETRIS to a new backbone. The diagnostic prior btb_t is computed from the same layer. All experiments are executed on a single NVIDIA A100 GPU. Layer choices, calibration, and alternative schedules are detailed in Appendices A.2.2 and A.3.3. 5.2 Main Results We report overall performance and efficiency across four compression rates on Qwen2-Audio in Table 1. Three key observations emerge from these results: AudioTETRIS maintains high accuracy even under heavy compression. At low compression (rate 0.36), AudioTETRIS matches or improves upon the uncompressed baseline on many benchmarks (4.52 WER, 58.04% IEMOCAP, 42.36% MMAU-mini vs. 4.61, 55.11%, 42.20% uncompressed). Even at rate 0.75, WER stays at 8.50 while IEMOCAP, MMAU-mini, and CoVoST2 remain within 2 points of uncompressed baselines, confirming that our method removes redundant tokens without disrupting essential lexical and semantic transitions. Performance gaps widen as compression grows aggressive. At rate 0.36, several baselines remain competitive; by rate 0.75, baseline methods degrade sharply (random pruning WER climbs to 31.59), lacking a mechanism to safeguard acoustic transitions. AudioTETRIS outperforms the best baseline by over 1.2 WER points at the highest compression setting. ASR is inherently more vulnerable to token pruning. Fine-grained transcription (WER) degrades far faster under aggressive compression than high-level understanding (MMAU-mini) or emotion recognition (IEMOCAP): at rate 0.75, IEMOCAP and MMAU-mini stay nearly intact while WER roughly doubles. High-level tasks rely on coarse acoustic features and tolerate token dropout, whereas exact lexical recovery needs fine-grained transition preservation - a vulnerability that speech-specific methods (AudioTETRIS, SpeechPrune) mitigate far better than general-purpose vision baselines or non-speech heuristics. 5.3 Efficiency Analysis Pruning within the LLM backbone yields direct computational savings, dropping total TFLOPs from 7.89 to 4.32 at a 0.75 compression rate, with latency falling from 1.31 s to 0.62 s over the same range. In-LM methods (AudioTETRIS, SparseVLM) incur slightly higher TFLOPs than single-shot methods (SpeechPrune, LTBM) since early layers still process the full, unpruned sequence before the first pruning stage; this difference is minimal relative to AudioTETRIS’s accuracy gains. Real-world latency scales somewhat less than the FLOP drop due to RoPE reindexing and KV-cache trimming overhead at intermediate layers (Appendix A.1.4). 5.4 Generalization Across Backbones To test whether AudioTETRIS’s gains are specific to Qwen2-Audio’s architecture, we repeat the main comparison on Phi-4-Multimodal and Audio Flamingo 3 at compression ratio 0.510.51 (Figure 4). On both backbones, AudioTETRIS remains within 1 point of uncompressed WER and attains the strongest or near-strongest accuracy on MMAU-mini, while baselines that match it on one metric (e.g., WER) fall sharply behind on others (e.g., SpeechPrune loses over 17 BLEU points on CoVoST2); Full numbers are given in Table 4 (Phi-4-Multimodal) and Table 5 (Audio Flamingo 3). These results suggest the phonetic transition prior transfers across encoders with different subsampling rates and architectures (Phi-4-Multimodal’s Conformer encoder vs. the Whisper-based encoders of Qwen2-Audio and Audio Flamingo 3), consistent with encoder-state transitions being a general property of speech representations rather than an artifact of one backbone. Extended results across all four compression ratios and benchmarks are in Appendix A.3.1, where similar patterns hold. Table 1: Performance and efficiency comparison across compression ratios on Qwen2Audio evaluated on LibriSpeech (ASR), IEMOCAP (Emotion Recognition), MMAU-mini (Audio Understanding, MCQ), and CoVoST2 (Speech Translation). WER (%) is lower-is-better (↓ ); Accuracy (%) and BLEU are higher-is-better (↑ ). Baselines: SpeechPrune (Lin et al., 2025), SparseVLM (Zhang et al., 2025b), LTBM (Luo et al., 2026). Method LibriSpeech WER ↓ IEMOCAP Acc. ↑ MMAU-mini Acc. ↑ CoVoST2 BLEU ↑ Latency (s) TFLOPs No Compression 4.61±0.124.61^± 0.12 55.11±1.0655.11^± 1.06 42.20±0.9642.20^± 0.96 33.03±0.7233.03^± 0.72 1.311.31 7.897.89 Compression Ratio 0.36 Random 9.91±1.679.91^± 1.67 53.10±0.3753.10^± 0.37 40.18±0.2340.18^± 0.23 29.10±0.0229.10^± 0.02 1.151.15 6.936.93 SpeechPrune 6.75±1.446.75^± 1.44 56.43±0.3556.43^± 0.35 41.17±0.6041.17^± 0.60 31.34±0.0731.34^± 0.07 1.211.21 6.996.99 SparseVLM†‡ 8.40±1.088.40^± 1.08 52.37±0.2152.37^± 0.21 41.09±0.7341.09^± 0.73 31.38±0.1231.38^± 0.12 1.161.16 7.287.28 LTBM 5.90±0.485.90^± 0.48 55.90±0.2755.90^± 0.27 40.67±0.6740.67^± 0.67 29.97±0.0629.97^± 0.06 1.251.25 6.936.93 AudioTETRIS† 4.52±0.214.52^± 0.21 58.04±0.8458.04^± 0.84 42.36±0.2142.36^± 0.21 31.96±0.0431.96^± 0.04 1.191.19 7.337.33 Compression Ratio 0.51 Random 14.02±0.3314.02^± 0.33 52.08±0.4952.08^± 0.49 39.20±0.4339.20^± 0.43 26.95±0.0126.95^± 0.01 0.970.97 5.615.61 SpeechPrune 6.52±1.146.52^± 1.14 57.39±0.7457.39^± 0.74 41.83±0.0641.83^± 0.06 31.16±0.1531.16^± 0.15 1.141.14 5.675.67 SparseVLM†‡ 9.13±1.159.13^± 1.15 47.10±0.6947.10^± 0.69 38.80±0.2738.80^± 0.27 31.17±0.0931.17^± 0.09 1.151.15 6.106.10 LTBM 6.92±0.046.92^± 0.04 54.58±0.4254.58^± 0.42 40.33±0.4040.33^± 0.40 29.55±0.0329.55^± 0.03 1.151.15 5.625.62 AudioTETRIS† 5.77±0.335.77^± 0.33 57.03±0.4157.03^± 0.41 42.91±0.3642.91^± 0.36 31.45±0.0631.45^± 0.06 1.001.00 6.136.13 Compression Ratio 0.64 Random 24.08±0.2624.08^± 0.26 50.22±1.1050.22^± 1.10 39.23±0.4139.23^± 0.41 24.64±0.0224.64^± 0.02 0.840.84 4.474.47 SpeechPrune 8.62±0.078.62^± 0.07 56.10±1.0956.10^± 1.09 40.30±0.6940.30^± 0.69 30.12±0.0530.12^± 0.05 0.990.99 4.534.53 SparseVLM†‡ 12.40±0.5512.40^± 0.55 47.27±1.7447.27^± 1.74 37.66±0.6637.66^± 0.66 30.79±0.1230.79^± 0.12 0.950.95 5.085.08 LTBM 8.60±0.188.60^± 0.18 53.38±0.8653.38^± 0.86 40.70±0.7840.70^± 0.78 29.47±0.0629.47^± 0.06 1.031.03 4.494.49 AudioTETRIS† 6.62±0.296.62^± 0.29 57.90±0.7657.90^± 0.76 41.83±0.1541.83^± 0.15 31.24±0.1131.24^± 0.11 0.860.86 5.265.26 Compression Ratio 0.75 Random 31.59±0.2431.59^± 0.24 47.67±0.7147.67^± 0.71 38.57±1.1638.57^± 1.16 22.82±0.0122.82^± 0.01 0.610.61 3.523.52 SpeechPrune 9.75±0.109.75^± 0.10 54.17±0.9254.17^± 0.92 40.93±0.3440.93^± 0.34 25.77±0.0525.77^± 0.05 0.690.69 3.583.58 SparseVLM†‡ 17.30±0.7017.30^± 0.70 46.10±3.0546.10^± 3.05 35.83±0.1235.83^± 0.12 30.66±0.0230.66^± 0.02 0.710.71 4.234.23 LTBM 10.58±0.4210.58^± 0.42 47.84±1.0547.84^± 1.05 39.23±0.4539.23^± 0.45 24.26±0.0524.26^± 0.05 0.700.70 3.523.52 AudioTETRIS† 8.50±0.188.50^± 0.18 55.34±1.2555.34^± 1.25 41.46±0.3241.46^± 0.32 31.33±0.0731.33^± 0.07 0.620.62 4.324.32 † In-LM pruning (token elimination performed inside the LLM backbone). ‡ Originally developed for the vision modality, adapted here for speech/audio. Figure 4: Comparison of LibriSpeech WER (↓ ) and MMAU-mini accuracy (↑ ) at compression ratio 0.51, for Phi-4-Multimodal and Audio Flamingo 3. Error bars show standard deviation across runs. AudioTETRIS is highlighted in the darker shade. 6 Ablation Studies We isolate the contribution of each component of AudioTETRIS’s scoring function and validate the choice of distance metric underlying the transition score §3.2. 6.1 Distance Metric for the Transition Score The transition signal dtd_t requires a distance function to measure the transition magnitude between adjacent encoder hidden states. We compare four standard choices: Euclidean (L2), cosine, Manhattan (L1), and Chebyshev (L∞) distance, holding all other hyperparameters fixed at a compression ratio of 0.510.51. Results are summarized in Figure 5. Figure 5: WER/accuracy/BLEU across distance metrics for the transition score, at compression ratio 0.510.51. Error bars show standard deviation over runs. We find that L2 distance is the strongest choice on all four benchmarks, with the largest gap on ASR (cosine increases WER by over 50%: 9.04%9.04\% vs. 5.77%5.77\%); hence we use it as the default throughout. 6.2 Component and Stage Ablation Analysis The fused score st(ℓ)=p~t(ℓ)⋅wts_t^( )= p_t^( )· w_t (Eq. 5) combines the depth-adaptive attention term with the acoustic prior. Figure 6 isolates each stage’s contribution against the full pipeline at compression ratio 0.510.51; extended results across all ratios and remaining benchmarks are in Table 6 (Appendix A.3.2). Figure 6: Ablation of AudioTETRIS stages at compression ratio 0.510.51. We find that both stages combined consistently outperforms either stage alone across all ratios and benchmarks, with the gap widening sharply under aggressive compression (at ratio 0.750.75, Stage 2 alone degrades to 15.56%15.56\% WER vs. 8.50%8.50\% combined). This reflects complementary signals: the phonetic prior wtw_t preserves phonetically contiguous spans, while the attention term targets task relevance - most visible on LibriSpeech, where ASR’s sensitivity to fine-grained transitions makes the two-stage advantage largest. 7 Conclusion We presented AudioTETRIS, a training-free framework for compressing audio tokens in LALMs. By leveraging the audio encoder’s hidden-state dynamics as a phoneme-transition prior and fusing it with depth-adaptive decoder attention, AudioTETRIS preserves acoustically rich and semantically critical regions without requiring auxiliary training or fine-tuning. Experiments across Qwen2-Audio, Phi-4-Multimodal, and Audio Flamingo 3, demonstrate that AudioTETRIS retains near-lossless performance at light compression, while widening its advantage over training-free baselines as compression gets aggressive. Furthermore, our findings highlight that ASR is highly sensitive to token loss making speech-aware structural priors essential for exact lexical recovery. Ultimately, this work highlights a broader principle for multimodal architecture design and token-compression methods: intermediate representations computed by modality-specific encoders contain valuable structural geometry that should be propagated forward rather than discarded at the LLM interface. Limitations AudioTETRIS’s acoustic prior assumes speech-encoder hidden states exhibit large-norm transitions at phonetic boundaries - validated for Whisper-based encoders (Qwen2-Audio, Audio Flamingo 3) and the Conformer-based Phi-4-Multimodal encoder, but not confirmed for encoders trained with different objectives or aggressive temporal downsampling (e.g., HTSAT-style audio-event encoders). Second, while the early pruning layer is fixed by construction, the later crossover layer ℓ† requires a short backbone-specific calibration pass (Appendix A.2.2) - a one-time cost, but the method is not fully zero-shot to unseen architectures without it. Third, because pruning happens progressively inside the decoder rather than once before prefill, AudioTETRIS carries a small TFLOP overhead relative to single-shot methods (§5.3), though this is largely offset by its accuracy gains. Finally, our evaluation spans four English-source benchmarks; results on noisier, or multilingual audio remain untested, and pushing compression beyond the rates studied here would likely erode lexical accuracy first, given ASR’s sensitivity to aggressive compression. Acknowledgements Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Commission. Neither the European Union nor the European Commission can be held responsible for them. This work was supported by the European Union’s Horizon Europe research and innovation programme under grant agreement No 101214398 (ELLIOT). This publication is part of project 2026.008 of the Computing Time National Computing Facilities program which is financed by the Dutch Research Council (NWO) under the grant https://doi.org/10.61686/WBFDE48237. Reproducibility Statement AudioTETRIS is training-free, so no training procedure, optimizer, or learning-rate schedule needs to be specified for reproduction. All components needed to reproduce our results are given in the main text and appendix: the phoneme transition score (Eq. 1-2), the depth-adaptive attention score and its automated crossover-layer discovery procedure (Eq. 3-4, Appendix A.2.2). A one-time, per-backbone calibration pass (Appendix A.2.2) locates the crossover layer ℓ† from attention statistics; this involves no gradient computation or parameter updates. The full fused-score pruning schedule are summarized as pseudocode in Algorithms 1 and 2 (Appendix A.2.1). Pruning layers, per-stage retention rates, and the closed-form multi-stage schedule are specified in §5.1 and Appendix A.3.3 (Eq. 9, Table 7). Backbone checkpoints, exact dataset splits, and their licenses are listed in Appendix A.1.1 (Tables 2, 3); alignment sources used for the boundary-correlation analysis (§4) are described in Appendix A.2.4. We will release our full implementation, including the pruning modules described in Algorithms 1 and 2 and the evaluation scripts used to produce Tables 1-5, at https://github.com/ShreyaSajid/AudioTETRIS. References Aylett and Turk (2004) M. Aylett and A. Turk The smooth signal redundancy hypothesis: a functional explanation for relationships between redundancy, prosodic prominence, and duration in spontaneous speech. Language and Speech 47 (1), p. 31–56. External Links: Document Cited by: §1. Behera et al. (2024) S. R. Behera, A. Dhiman, K. Gowda, and A. S. Narayani FastAST: accelerating audio spectrogram transformer via token merging and cross-model knowledge distillation. In Interspeech 2024, p. 4303–4307. External Links: Document, Link Cited by: §1, §2. Bhati et al. (2025) S. Bhati, S. Thomas, H. Kuehne, R. Feris, and J. Glass Towards audio token compression in large audio language models. arXiv preprint arXiv:2511.20973. Cited by: §1, §2. Bolya et al. (2023) D. Bolya, C. Fu, X. Dai, P. Zhang, C. Feichtenhofer, and J. Hoffman Token merging: your vit but faster. In International Conference on Learning Representations (ICLR), Cited by: §2. Busso et al. (2008) C. Busso, M. Bulut, C. Lee, A. Kazemzadeh, E. Mower Provost, S. Kim, J. Chang, S. Lee, and S. Narayanan IEMOCAP: interactive emotional dyadic motion capture database. Language Resources and Evaluation 42, p. 335–359. External Links: Document Cited by: §A.2.4, §A.2.4, Table 3, §5.1. Chen et al. (2024) L. Chen, H. Zhao, T. Liu, S. Bai, J. Lin, C. Zhou, and B. Chang An image is worth 1/2 tokens after layer 2: plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision (ECCV), p. 19–35. External Links: Document Cited by: §2. Chu et al. (2024) Y. Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y. Leng, Y. Lv, J. He, and J. Lin Qwen2-audio technical report. External Links: 2407.10759, Link Cited by: Figure 7, Table 2, §1, §1, §1, §2, §4, §5.1. Chu et al. (2023) Y. Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou Qwen-audio: advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919. Cited by: §1. CorentinJ (2019) CorentinJ LibriSpeech alignments. Note: https://github.com/CorentinJ/librispeech-alignmentsGitHub repository Cited by: §A.2.4, §4. Dao (2024) T. Dao FlashAttention-2: faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), Cited by: §A.1.4, §A.1.4. Deshmukh et al. (2023) S. Deshmukh, B. Elizalde, R. Singh, and H. Wang Pengi: an audio language model for audio tasks. In Advances in Neural Information Processing Systems, Vol. 36, p. 18090–18108. Cited by: §2. Furui (1986a) S. Furui On the role of spectral transition for speech perception. The Journal of the Acoustical Society of America 80 (4), p. 1016–1025. External Links: Document Cited by: §1, §1. Furui (1986b) S. Furui Speaker-independent isolated word recognition using dynamic features of speech spectrum. IEEE Transactions on Acoustics, Speech, and Signal Processing 34 (1), p. 52–59. External Links: Document Cited by: §3.2, §4. Ghosh et al. (2025a) S. Ghosh, A. Goel, J. Kim, S. Kumar, Z. Kong, S. Lee, C. H. Yang, R. Duraiswami, D. Manocha, R. Valle, and B. Catanzaro Audio flamingo 3: advancing audio intelligence with fully open large audio language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Document Cited by: Figure 7, Table 2, §1, §1, §1, §2, §4, §5.1. Ghosh et al. (2025b) S. Ghosh, Z. Kong, S. Kumar, S. Sakshi, J. Kim, W. Ping, R. Valle, D. Manocha, and B. Catanzaro Audio flamingo 2: an audio-language model with long-audio understanding and expert reasoning abilities. In Proceedings of the 42nd International Conference on Machine Learning, A. Singh, M. Fazel, D. Hsu, S. Lacoste-Julien, F. Berkenkamp, T. Maharaj, K. Wagstaff, and J. Zhu (Eds.), Vol. 267, p. 19358–19405. Cited by: §2. Gong et al. (2026) C. Gong, D. Wang, Z. Wei, Y. Guo, H. Zhu, and J. Chen EchoingPixels: aliasing-resistant joint token reduction for audio-visual llms. In Forty-third International Conference on Machine Learning, Cited by: §1, §2. He et al. (2025) P. He, Z. Wen, Y. Wang, Y. Wang, X. Liu, J. Huang, Z. Lei, Z. Gu, X. Jin, J. Yang, K. Li, Z. Liu, W. Li, C. A. Wang, C. He, and L. Zhang AudioMarathon: a comprehensive benchmark for long-context audio understanding and efficiency in audio llms. arXiv preprint arXiv:2510.07293. Cited by: §2. Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. In ICLR, Cited by: §2. Jung et al. (2026) C. Jung, Y. Jang, S. Lee, and J. S. Chung FastAV: efficient token pruning for audio-visual large language model inference. In ICASSP 2026 - 2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), External Links: Document Cited by: §1, §2. Kavaki and Mandel (2025) H. S. Kavaki and M. I. Mandel Audio sparse-transformer for speech classification. In ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), External Links: Document Cited by: §2. Li et al. (2023) Y. Li, Y. Wu, J. Li, and S. Liu Accelerating Transducers through Adjacent Token Merging. In Interspeech 2023, p. 1379–1383. External Links: Document, ISSN 2958-1796 Cited by: §2. Libera et al. (2026) L. D. Libera, C. Subakan, and M. Ravanelli Beyond fixed frames: dynamic character-aligned speech tokenization. External Links: 2601.23174, Link Cited by: §1. Lin et al. (2025) Y. Lin, Y. Fu, J. Zhang, Y. Liu, J. Zhang, J. Sun, H. Li, and Y. Chen SpeechPrune: context-aware token pruning for speech information retrieval. In 2025 IEEE International Conference on Multimedia and Expo (ICME), Cited by: §A.1.4, §1, §1, §2, §5.1, Table 1. Luo et al. (2026) J. Luo, X. Liang, and H. Hu Locality matters for training-free audio token compression in audio-language models. arXiv preprint arXiv:2605.25179. Cited by: §2, §5.1, Table 1. McAuliffe et al. (2017) M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger Montreal forced aligner: trainable text-speech alignment using kaldi. In Proceedings of the 18th Annual Conference of the International Speech Communication Association (INTERSPEECH 2017), p. 498–502. External Links: Document Cited by: §A.2.4. Microsoft et al. (2025) Microsoft, A. Abouelenin, A. Ashfaq, A. Atkinson, H. Awadalla, N. Bach, J. Bao, A. Benhaim, M. Cai, V. Chaudhary, C. Chen, D. Chen, D. Chen, J. Chen, W. Chen, Y. Chen, Y. Chen, Q. Dai, X. Dai, R. Fan, M. Gao, M. Gao, A. Garg, A. Goswami, J. Hao, A. Hendy, Y. Hu, X. Jin, M. Khademi, D. Kim, Y. J. Kim, G. Lee, J. Li, Y. Li, C. Liang, X. Lin, Z. Lin, M. Liu, Y. Liu, G. Lopez, C. Luo, P. Madan, V. Mazalov, A. Mitra, A. Mousavi, A. Nguyen, J. Pan, D. Perez-Becker, J. Platin, T. Portet, K. Qiu, B. Ren, L. Ren, S. Roy, N. Shang, Y. Shen, S. Singhal, S. Som, X. Song, T. Sych, P. Vaddamanu, S. Wang, Y. Wang, Z. Wang, H. Wu, H. Xu, W. Xu, Y. Yang, Z. Yang, D. Yu, I. Zabir, J. Zhang, L. L. Zhang, Y. Zhang, and X. Zhou Phi-4-mini technical report: compact yet powerful multimodal language models via mixture-of-loras. External Links: 2503.01743, Link Cited by: Figure 7, Table 2, §1, §2, §4, §5.1. Panayotov et al. (2015) V. Panayotov, G. Chen, D. Povey, and S. Khudanpur LibriSpeech: an asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 5206–5210. Cited by: §A.2.2, Table 3, §4, §5.1. Radford et al. (2023) A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever Robust speech recognition via large-scale weak supervision. ICML. Cited by: §1, §1, §2, §4. Sakshi et al. (2025) S. Sakshi, U. Tyagi, S. Kumar, A. Seth, R. Selvakumar, O. Nieto, R. Duraiswami, S. Ghosh, and D. Manocha MMAU: a massive multi-task audio understanding and reasoning benchmark. In The Thirteenth International Conference on Learning Representations (ICLR), Cited by: Table 3, §5.1. Shang et al. (2025) Y. Shang, M. Cai, B. Xu, Y. J. Lee, and Y. Yan LLaVA-prumerge: adaptive token reduction for efficient large multimodal models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 22857–22867. Cited by: §2. Shao et al. (2026) K. Shao, K. Tao, K. Zhang, S. Feng, M. Cai, Y. Shang, H. You, C. Qin, Y. Sui, and H. Wang A survey of token compression for efficient multimodal large language models. Transactions on Machine Learning Research. External Links: Link Cited by: §2. Stevens (2002) K. N. Stevens Toward a model for lexical access based on acoustic landmarks and distinctive features. The Journal of the Acoustical Society of America 111 (4), p. 1872–1891. External Links: Document Cited by: §1, §1. Tang et al. (2024) C. Tang, W. Yao, G. Zhou, B. Chen, S. Deng, Z. Liu, J. Wang, Z. Zhou, X. Zhang, Z. Wan, et al. SALMONN: towards generic hearing abilities for large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2. Tay et al. (2022) Y. Tay, M. Dehghani, D. Bahri, and D. Metzler Efficient transformers: a survey. ACM Comput. Surv. 55 (6). External Links: ISSN 0360-0300, Link, Document Cited by: §1. Wang et al. (2025a) B. Wang, X. Zou, G. Lin, S. Sun, Z. Liu, W. Zhang, Z. Liu, A. Aw, and N. F. Chen AudioBench: a universal benchmark for audio large language models. In Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 4297–4316. External Links: Link, Document Cited by: §A.1.2, §1. Wang et al. (2021) C. Wang, A. Conneau, J. Gu, and J. Pino CoVoST 2 and Massively Multilingual Speech Translation. In Proc. Interspeech 2021, p. 4234–4238. External Links: Document, ISSN 2958-1796 Cited by: §A.2.4, Table 3, §5.1. Wang et al. (2025b) J. Wang, Y. Zhao, S. Wang, H. Lin, S. Wang, S. Feng, and D. Jiang Say more with less: variable-frame-rate speech tokenization via adaptive clustering and implicit duration coding. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 25310–25318. External Links: Link Cited by: §1. Xing et al. (2025) L. Xing, Q. Huang, X. Dong, J. Lu, P. Zhang, Y. Zang, Y. Cao, C. He, J. Wang, F. Wu, and D. Lin PyramidDrop: accelerating your large vision-language models via pyramid visual redundancy reduction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2. Yang et al. (2024) Q. Y. Yang, H. Wang, Z. Li, S. Wang, S. Zhang, M. Liu, J. Guo, Y. Zhou, C. Su, Z. Zhang, C. Li, and W. Xu AIR-bench: benchmarking large audio-language models via generative comprehension. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 1966–1986. External Links: Link Cited by: §1. Yang et al. (2025) S. Yang, Y. Chen, Z. Tian, C. Wang, J. Li, B. Yu, and J. Jia VisionZip: longer is better but not necessary in vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2. Zhang et al. (2025a) X. Zhang, F. Fang, P. Gao, B. Qin, B. Ahmed, and J. Epps Distinctive feature codec: an adaptive efficient speech representation for depression detection. External Links: 2505.18516, Link Cited by: §1. Zhang et al. (2026) X. Zhang, Y. Li, H. Zhang, S. Han, H. Liu, Q. Zhang, B. Ahmed, and J. Epps The wer trap: shattering the illusion of unified tokens in speech language models. arXiv preprint arXiv:2605.29209. Cited by: §1, §1. Zhang et al. (2025b) Y. Zhang, C. Fan, J. Ma, W. Zheng, T. Huang, K. Cheng, D. Gudovskiy, T. Okuno, Y. Nakata, K. Keutzer, et al. SparseVLM: visual token sparsification for efficient vision-language model inference. In International Conference on Machine Learning, Cited by: §A.1.3, §A.1.4, §2, §5.1, Table 1. Appendix A Appendix A.1 Implementation Details A.1.1 Backbones and Checkpoints Table 2 lists the exact checkpoints and licenses for the three backbones evaluated in this work. All experiments use unmodified pretrained weights; the only backbone-side code changes are load-time monkey-patches to the decoder forward pass implementing AudioTETRIS’s pruning (§3). No checkpoint weights were altered or fine-tuned. Table 2: Pretrained checkpoints used in this work. Backbone Checkpoint License Qwen2-Audio (Chu et al., 2024) Qwen/Qwen2-Audio-7B-Instruct Apache 2.0 Phi-4-Multimodal (Microsoft et al., 2025) microsoft/Phi-4-multimodal-instruct MIT Audio Flamingo 3 (Ghosh et al., 2025a) nvidia/audio-flamingo-3-hf NVIDIA OneWay Noncommercial License* * Non-commercial research use only; see checkpoint card for full terms. A.1.2 Datasets, Splits, and Evaluation Table 3 lists the datasets used for evaluation, the exact splits, and their licensing. We accessed all four via the AudioLLMs organization’s re-packaged mirrors on the Hugging Face Hub, used throughout the AudioBench evaluation harness (Wang et al., 2025a), rather than the original distribution channels for each corpus. For the CoVoST2 phoneme alignments generated for Appendix A.2.4, these are derivative annotations produced by us using the Montreal Forced Aligner and are not redistributed third-party material. Table 3: Evaluation datasets, splits, and licensing. Dataset Split used Original license LibriSpeech (test-other) (Panayotov et al., 2015) AudioLLMs/librispeech_test_other, test C BY 4.0 IEMOCAP (Busso et al., 2008) AudioLLMs/iemocap_emotion_recognition, test Custom non-commercial research license (USC SAIL) MMAU-mini (Sakshi et al., 2025) AudioLLMs/MMAU-mini, test Not explicitly stated by source CoVoST2 (En→ ) (Wang et al., 2021) AudioLLMs/covost2_en_zh_test, test C0 (per original release); Hub mirror tagged C BY-NC 4.0 A.1.3 SparseVLM Adaptation to Fixed-Rate Speech Pruning SparseVLM (Zhang et al., 2025b) originally proposed an adaptive, dynamic pruning schedule based on the rank of the visual-text attention matrix. However, its public release22 2 https://github.com/Gumpest/SparseVLMs, accessed May 2026, commit 87fe4319430e079a778c25d4ddf4588a3c4038ca 2025-12-22. replaces this dynamic controller with static token-retention schedules. Because the dynamic rank-based ratio is absent from all usable releases (main, v1.5, v2.0), we could not evaluate SparseVLM’s originally described adaptive deployment directly. Instead, we adapt its public attention-guided scoring mechanism to our fixed-rate setup for a direct, controlled comparison. Adaptation. We retain SparseVLM’s core scoring mechanism - using the text raters, and text-token attention to rate audio token importance - and apply it at layers 1,15\1,15\ (Appendix A.2.2). At each stage, we keep the top fraction r of audio tokens by score to match our evaluated compression targets (0.36,0.51,0.64,0.750.36,0.51,0.64,0.75). This adaptation isolates the impact of SparseVLM’s attention-based importance metric under identical layer placements and compute budgets. A.1.4 Attention Backend for Score-Based Pruning AudioTETRIS and the attention-score-based baselines we compare against (SparseVLM Zhang et al. (2025b), SpeechPrune (Lin et al., 2025)) all derive token-importance scores from attention probabilities between query and audio tokens. Fused attention kernels such as FlashAttention-2 (Dao, 2024) and PyTorch’s scaled-dot-product attention (SDPA) do not materialize these probabilities: both return attn_weights=None, since the attention matrix is never explicitly formed. Eager attention is the only implementation that exposes the full [B,H,N(ℓ),N(ℓ)][B,H,N^( ),N^( )] probability tensor required for scoring. We therefore run the attention backend in eager mode only at the layers where token scores are computed, and restore the model’s configured kernel (FlashAttention-2 (Dao, 2024), in our setup) for the forward pass that produces the hidden states and KV cache used downstream. This substitution is local to the small set of scoring layers (§3) rather than applied model-wide, so the FlashAttention-2 fast path is preserved for the remainder of the network’s depth. We apply this identically across AudioTETRIS and all attention-score-based baselines, so the latency and FLOPs figures reported in §5.2 reflect the same hybrid execution path for every method and remain directly comparable. We note that this eager-attention fallback is a memory-bandwidth-bound cost that is not reflected in the TFLOPs column, which counts compute operations only; it is, however, fully captured in our reported wall-clock latency, since all latency numbers are measured end-to-end on the actual hybrid execution path (eager at scoring layers, FlashAttention-2 elsewhere). Because the eager fallback is confined to exactly two decoder layers per backbone (layer 1 and that backbone’s calibrated ℓ† ; §5.1), its contribution to total latency is small in practice, and it could be eliminated entirely in a production deployment with a custom fused kernel that computes only the row/column marginal attention sums our scoring requires, without materializing the full attention matrix - a one-time kernel-engineering cost rather than a potential structural limitation of the method. A.2 Method Details and Analysis A.2.1 Algorithmic Summary For reproducibility, Algorithms 1 and 2 summarize the full pipeline described in §3. Algorithm 1 is a one-time, offline calibration step run once per backbone to locate the crossover layer ℓ† (§A.2.2); Algorithm 2 is the training-free inference procedure applied per input, using ℓ† from calibration to determine which attention branch of Eq. 3 to use at each pruning layer. Algorithm 1 Crossover Layer Calibration (offline, run once per backbone) 1: Calibration set U (audio-query pairs), backbone with LdecL_dec decoder layers 2: Crossover layer ℓ† 3: for each utterance u∈u do 4: Run forward pass; record attention tensors Au(ℓ)A^( )_u for all layers ℓ=1,…,Ldec =1,…,L_dec 5: end for 6: for ℓ=1,…,Ldec =1,…,L_dec do 7: aintra(ℓ)←a_intra( )← mean attention mass from aQ_a rows onto audio tokens (Eq. 3, upper case), averaged over U 8: across(ℓ)←a_cross( )← mean attention mass from qQ_q rows onto audio tokens (Eq. 3, lower case), averaged over U 9: end for 10: ℓ†←minℓ:across(ℓ)≥aintra(ℓ) ← \ :a_cross( )≥ a_intra( )\ ⊳ Eq. 4 11: return ℓ† Algorithm 2 AudioTETRIS Inference (training-free, applied per input) 1: Audio encoder hidden states H=h1,…,hTH=\h_1,…,h_T\, pruning layers ℓ1,ℓ2\ _1, _2\ with ℓ1<ℓ†≤ℓ2 _1< ≤ _2 (in practice ℓ2=ℓ† _2= , the backbone’s calibrated crossover layer; §5.1), target retention fractions kℓ1,kℓ2\k_ _1,k_ _2\, decoder with LdecL_dec layers 2: Compressed decoder output 3: Stage 1: Acoustic Transition Prior (computed once, before decoding) 4: d1←0d_1← 0 5: for t=2,…,Tt=2,…,T do 6: dt←‖ht−ht−1‖22d_t←\|h_t-h_t-1\|_2^2 ⊳ Eq. 1 7: end for 8: d¯←1T∑i=1Tdi d← 1T _i=1^Td_i 9: for t=1,…,Tt=1,…,T do 10: wt←dt/d¯w_t← d_t/ d ⊳ Eq. 2 11: end for 12: Stage 2: Depth-Adaptive Pruning (during decoder forward pass) 13: ←1,…,TT←\1,…,T\ ⊳ surviving audio-token indices 14: for ℓ=1,…,Ldec =1,…,L_dec do 15: Run decoder layer ℓ on current sequence (audio tokens indexed by T, plus query tokens) 16: if ℓ∈ℓ1,ℓ2 ∈\ _1, _2\ then 17: if ℓ<ℓ† < then 18: p~t(ℓ)← p_t^( )← mean attention from audio-query rows aQ_a onto token t, ∀t∈∀ t ⊳ Eq. 3, intra-audio 19: else 20: p~t(ℓ)← p_t^( )← mean attention from text-query rows qQ_q onto token t, ∀t∈∀ t ⊳ Eq. 3, cross-modal 21: end if 22: st(ℓ)←p~t(ℓ)⋅wt,∀t∈s_t^( )← p_t^( )· w_t, ∀ t ⊳ Eq. 5 23: ←T← indices of the top ⌈kℓ⋅||⌉ k_ ·|T| tokens in T ranked by st(ℓ)s_t^( ) 24: Evict pruned tokens from hidden states and KV cache; update RoPE position indices for T 25: end if 26: end for 27: return final decoder output A.2.2 Pruning Layer Selection AudioTETRIS prunes at two decoder layers (§5.1), and the two are chosen by different criteria because they serve different roles. The early layer is about saving compute: it is fixed by design as the earliest depth at which the attention term exists, and pruning there is viable because the acoustic prior compensates for weak early attention. The late layer ℓ† is about task relevance: it is calibrated per backbone from attention crossover statistics to locate where cross-modal grounding takes over (§A.2.2). We motivate each choice separately below. Early Pruning Layer. Because self-attention cost accrues at every subsequent decoder layer, the earlier a token is removed, the larger the cumulative FLOP reduction across the rest of the forward pass. We therefore place the first pruning stage at layer 1 - the earliest layer at which the depth-adaptive attention score p~t(ℓ) p_t^( ) (Eq. 3) is defined, since it requires at least one decoder self-attention computation to exist. Deferring pruning to a deeper early layer might yield a marginally sharper intra-audio attention signal, but this benefit is outweighed by the compute lost across every additional layer the full, unpruned token sequence has to pass through. Early pruning works well for AudioTETRIS because the layer-1 score does not rely on attention alone. The fused score st(1)=p~t(1)⋅wts_t^(1)= p_t^(1)· w_t (Eq. 5) combines the attention-based importance p~t(1) p_t^(1), computed from a single decoder layer, with the acoustic transition prior wtw_t (Eq. 2). The prior comes directly from the audio encoder’s hidden states, so it is already fully informative before the decoder even starts (§4). The prior’s independence from decoder depth matters because attention alone is a weak signal at layer 1, where the decoder has not yet built up much context. The acoustic prior compensates for this weakness, making the fused score far more reliable at this depth than attention alone would be. This reliability is what lets us prune aggressively at layer 1 without the accuracy collapse that a purely attention-based method suffers. Our component ablation supports this conclusion (§6.2, Table 6): ‘Only Stage 2’ (attention alone) degrades much faster as compression increases than either ‘Only Stage 1’ or the full fused score. Late Pruning Layer: Crossover Calibration. Section 3.3 defines the crossover layer ℓ† (Eq. 4) as the first decoder depth at which cross-modal attention mass overtakes intra-audio attention mass, and uses it to set the layer partition ℒearly,ℒlater\L_early,L_later\ for the depth-adaptive attention score p~t(ℓ) p_t^( ) (Eq. 3). Here we describe the calibration procedure used to estimate ℓ† and report the resulting per-layer attention statistics for each backbone. Calibration set. We estimate ℓ† from 150 utterances drawn from LibriSpeech test-other (Panayotov et al., 2015), disjoint from the utterances used for downstream evaluation in §5.2. For each calibration utterance, we run a forward pass through the target backbone and record the full decoder attention tensors A(ℓ)A^( ) at every layer, with no gradient updates or parameter changes. Calibration takes under 4 minutes on a single A100 and is amortized across all subsequent inputs for that backbone. Per-layer attention mass. For each decoder layer ℓ , we define the mean intra-audio and cross-modal attention mass, restating the two branches of Eq. 3 without the depth partition: aintra(ℓ) a_intra( ) =1||∑u∈1H⋅|a|⋅Tu∑h=1H∑i∈a∑t=1TuAh,i,t(ℓ), = 1|U| _u 1H·|Q_a|· T_u _h=1^H _i _a _t=1^T_uA^( )_h,i,t, (7) across(ℓ) a_cross( ) =1||∑u∈1H⋅|q|⋅Tu∑h=1H∑i∈q∑t=1TuAh,i,t(ℓ), = 1|U| _u 1H·|Q_q|· T_u _h=1^H _i _q _t=1^T_uA^( )_h,i,t, (8) where U is the calibration set, TuT_u is the number of audio tokens in utterance u, and aQ_a, qQ_q are the audio-query and text-query row index sets as in Eq. 3. The crossover layer is then ℓ†=minℓ:across(ℓ)≥aintra(ℓ) = \ :a_cross( )≥ a_intra( )\, matching Eq. 4. Figure 7: Per-layer intra-audio (aintra(ℓ)a_intra( )) and cross-modal (across(ℓ)a_cross( )) attention mass on the LibriSpeech test-other calibration subset, for Qwen2-Audio (Chu et al., 2024), Phi-4-Multimodal (Microsoft et al., 2025), and Audio Flamingo 3 (Ghosh et al., 2025a) The dashed vertical line marks the identified crossover layer ℓ† for each backbone. Intra-audio attention dominates in early layers and is progressively overtaken by cross-modal attention as depth increases, consistent with Figure 2. Results. As shown in Figure 7, all three backbones exhibit the same qualitative pattern: intra-audio attention mass is highest in early layers and declines monotonically (on average) with depth, while cross-modal attention mass rises correspondingly, crossing over in the middle third of the network. The calibration procedure identifies ℓ†=15 =15 for Qwen2-Audio, consistent with the pruning schedule used in §5.1, ℓ†=13 =13 for Phi-4-Multimodal, and ℓ†=18 =18 for Audio Flamingo 3. The crossover layer therefore shifts across backbones with differing encoder-decoder depth and subsampling rates. A.2.3 Transition Prior Computation Pipeline Figure 8 illustrates the end-to-end computation of the phoneme transition score btb_t and its relationship to ground-truth phoneme transitions, using a toy example spanning the phonemes /s/-/t/-/r/-/iy/-/m/. The raw derivative dtd_t (Eq. 1) is computed from consecutive encoder hidden states, producing sparse spikes near true phonetic transitions but with substantial token-to-token noise. Gaussian smoothing (Eq. 6) converts this into the continuous transition prior btb_t, which we correlate against both the discrete ground-truth transition indicator yhardy_hard and its Gaussian-smoothed counterpart ysofty_soft to obtain rhardr_hard and rsoftr_soft respectively. Figure 8: Schematic of the phoneme-transition-aware prior pipeline. Top: audio waveform with forced-alignment phoneme segments. Second row: per-token encoder hidden states hth_t, colored by phoneme membership. Third row: raw derivative dtd_t (bars) and the Gaussian-smoothed transition score btb_t (line). Bottom: ground-truth transition labels, shown as discrete spikes yhardy_hard and their Gaussian-smoothed form ysofty_soft, used to compute rhardr_hard and rsoftr_soft. A.2.4 Boundary Correlation on CoVoST2 and IEMOCAP Section 4 reports boundary-correlation results on LibriSpeech (Figure 3), using the MFA-based forced alignments made available through (CorentinJ, 2019). Here we repeat the same analysis on our two other evaluation datasets, CoVoST2 (speech translation) (Wang et al., 2021) and IEMOCAP (emotion recognition) (Busso et al., 2008), to confirm that the mid-network correlation peak is not an artifact of LibriSpeech’s read-speech domain. Alignment sources. IEMOCAP ships with official word-, syllable-, and phoneme-level forced alignments as part of its dataset release, generated with Sphinx-I (Busso et al., 2008); we use these directly. No comparable off-the-shelf alignment resource exists for CoVoST2, so we generated phoneme alignments ourselves using the Montreal Forced Aligner (MFA) (McAuliffe et al., 2017) on CoVoST2’s English-source audio and transcripts, matching the alignment methodology used for the LibriSpeech release to keep the comparison consistent across datasets.33 3 IEMOCAP’s original release notes that alignment accuracy was screened but not formally evaluated by the corpus authors; we flag this as a caveat on alignment quality relative to the MFA-based LibriSpeech and CoVoST2 alignments used elsewhere in this work. For both datasets we recompute btb_t per layer and correlate against the same yhardy_hard/ysofty_soft construction as in Section 4, holding σ=2σ=2 fixed. As shown in Figure 9, both datasets reproduce the mid-network peak observed on LibriSpeech across all three backbones, with rsoftr_soft peaking in the same layer range identified by the sweet-spot shading (shared with Figure 3). Peak rsoftr_soft is consistently higher on CoVoST2 than on IEMOCAP for every backbone (e.g., Qwen2-Audio: 0.670.67 vs. 0.600.60; Audio Flamingo 3: 0.610.61 vs. 0.580.58), which is consistent with our finding in Section 5.2 that fine-grained lexical tasks depend more heavily on precise boundary localization than paralinguistic classification does. Figure 9: Layer-wise boundary correlation (rsoftr_soft, rhardr_hard) across Qwen2-Audio, Phi-4-Multimodal, and Audio Flamingo 3, computed on CoVoST2 (top, MFA alignments generated in-house) and IEMOCAP (bottom, official released alignments). Shaded regions mark each backbone’s mid-network sweet spot, consistent with Figure 3. Dashed vertical lines and annotated values mark the peak layer for each curve. A.2.5 Sensitivity to the Smoothing Width σ Eq. 6 defines the transition prior btb_t by convolving the raw distance signal dtd_t with a Gaussian kernel of width σ, and we apply the same σ to smooth the discrete forced-alignment labels yhardy_hard into the continuous target ysofty_soft. In the main text we fix σ=2σ=2 (encoder tokens). Here we ablate this choice to characterize how rsoftr_soft and rhardr_hard vary as σ moves away from the selected smoothing width. We sweep σ∈0.5,1,2,3,4,5,6,8σ∈\0.5,1,2,3,4,5,6,8\ encoder tokens and recompute rsoftr_soft and rhardr_hard at each value, holding the layer fixed at the mid-network sweet spot identified in Section 4 for each backbone. Because Phi-4-Multimodal’s audio encoder subsamples more aggressively than Qwen2-Audio’s or Audio Flamingo’s, the same σ in token units corresponds to roughly twice the real-time smoothing window for Phi-4-Multimodal; Figure 10 therefore also reports σ in milliseconds on a secondary axis for each backbone. As shown in Figure 10, both rsoftr_soft and rhardr_hard consistently exhibit an inverted U-shaped curve across all backbones, peaking cleanly at σ=2σ=2 tokens. Performance degrades on either side: smaller values fail to account for context distribution, while larger values (σ>2σ>2) excessively blur distinct phonetic transitions. Figure 10: Sensitivity of the transition-prior correlation to the Gaussian smoothing width σ, on all backbones. Each panel sweeps σ (encoder tokens, with milliseconds on the secondary top axis) and reports rsoftr_soft and rhardr_hard at that backbone’s mid-network sweet-spot layer. The shaded band and dashed line mark σ=2σ=2, the value used throughout the main text. A.2.6 Qualitative Encoder Embedding Trajectories Figure 11 complements the quantitative diagnostics of Section 4 with a qualitative view of the same phenomenon. We project the per-token encoder hidden states hth_t of a single LibriSpeech test-other utterance into two dimensions with t-SNE at an early (layer 4), middle (layer 16), and late encoder layer (layer 28) of Qwen2-Audio, and trace the utterance’s token sequence through the projected space. Frames from other utterances are shown in grey to provide a shared spatial reference. At the middle layer, which coincides with the peak of both rsoftr_soft and kurtosis (Figure 3), tokens belonging to the same phoneme form compact clusters. At the early and late layers, tokens are less consolidated by phoneme identity and boundary frames are less clearly separated from within-segment frames, consistent with the lower correlation and kurtosis observed at those depths. Since t-SNE preserves local neighbourhoods but not global distances, we treat this figure as illustrative only. The quantitative evidence for the mid-network sweet spot is given by the correlation and kurtosis analyses in Section 4 and Appendix A.2.4. Figure 11: Qualitative analysis of Qwen2-Audio encoder token embedding trajectories for a single LibriSpeech utterance, projected via t-SNE and colored by phoneme identity, at early, middle, and late encoder depths. Colored points denote individual tokens where different colors represent different phonemes; Star markers indicate frames at phoneme transitions per forced alignment; Faint gray points show other utterances’ frames for spatial context. A.3 Additional Experiments A.3.1 Extended Results on Phi-4-Multimodal and Audio Flamingo 3 Section 5.2 and Figure 4 reports results on Phi-4-Multimodal and Audio Flamingo 3 at a single compression ratio (0.510.51) for brevity. Here we provide the extended results for both backbones across all four evaluated compression ratios (0.360.36, 0.510.51, 0.640.64, 0.750.75), reported in Table 4 for Phi-4-Multimodal and Table 5 for Audio Flamingo 3. Consistent with the Qwen2-Audio results in Table 1, AudioTETRIS maintains near-lossless performance across both additional backbones at low compression and shows a widening advantage over baselines as compression increases, indicating that the method generalizes across encoder-decoder architectures rather than being tuned to a single backbone. Table 4: Extended results for Phi-4-Multimodal across all compression ratios. Method LibriSpeech WER ↓ IEMOCAP Acc. ↑ MMAU-mini Acc. ↑ CoVoST2 BLEU ↑ Latency (s) TFLOPs No Compression 4.85±0.344.85^± 0.34 39.34±0.2939.34^± 0.29 54.0±0.8554.0^± 0.85 36.06±0.0736.06^± 0.07 1.151.15 1.4031.403 Compression Ratio 0.36 Random 5.64±1.925.64^± 1.92 39.00±0.5139.00^± 0.51 52.60±0.8952.60^± 0.89 19.05±0.0519.05^± 0.05 0.970.97 0.960.96 SpeechPrune 5.55±0.065.55^± 0.06 38.28±1.1038.28^± 1.10 53.70±0.4453.70^± 0.44 22.60±0.0722.60^± 0.07 1.011.01 0.960.96 SparseVLM†‡ 5.43±0.795.43^± 0.79 39.75±0.3639.75^± 0.36 52.70±0.4352.70^± 0.43 22.05±0.0722.05^± 0.07 1.021.02 1.001.00 LTBM 6.60±0.006.60^± 0.00 40.40±0.8440.40^± 0.84 54.40±0.0054.40^± 0.00 26.76±0.1926.76^± 0.19 1.001.00 0.960.96 AudioTETRIS† 4.84±1.034.84^± 1.03 40.93±0.5640.93^± 0.56 55.20±0.3155.20^± 0.31 31.60±0.3531.60^± 0.35 1.021.02 1.031.03 Compression Ratio 0.51 Random 7.99±0.687.99^± 0.68 38.03±0.9038.03^± 0.90 51.30±0.6451.30^± 0.64 16.34±0.0216.34^± 0.02 0.770.77 0.770.77 SpeechPrune 5.81±2.805.81^± 2.80 36.91±0.9936.91^± 0.99 53.20±0.9253.20^± 0.92 13.57±0.0113.57^± 0.01 0.800.80 0.780.78 SparseVLM†‡ 6.53±1.136.53^± 1.13 39.34±0.7239.34^± 0.72 52.70±1.0452.70^± 1.04 17.92±0.0817.92^± 0.08 0.810.81 0.840.84 LTBM 6.95±0.076.95^± 0.07 39.74±0.9539.74^± 0.95 53.60±0.0053.60^± 0.00 20.23±0.0220.23^± 0.02 0.790.79 0.770.77 AudioTETRIS† 5.78±1.495.78^± 1.49 39.90±0.2039.90^± 0.20 54.10±0.3254.10^± 0.32 31.13±0.3031.13^± 0.30 0.810.81 0.880.88 Compression Ratio 0.64 Random 8.49±6.068.49^± 6.06 35.84±0.2535.84^± 0.25 50.90±0.8950.90^± 0.89 25.29±0.0225.29^± 0.02 0.610.61 0.620.62 SpeechPrune 6.38±2.016.38^± 2.01 35.38±0.4035.38^± 0.40 51.90±0.8351.90^± 0.83 6.21±0.036.21^± 0.03 0.640.64 0.630.63 SparseVLM†‡ 7.53±1.097.53^± 1.09 39.04±0.5639.04^± 0.56 52.30±0.6352.30^± 0.63 15.67±0.0415.67^± 0.04 0.650.65 0.700.70 LTBM 6.80±0.466.80^± 0.46 38.97±0.7738.97^± 0.77 52.10±0.4552.10^± 0.45 15.24±0.1015.24^± 0.10 0.630.63 0.620.62 AudioTETRIS† 6.04±0.876.04^± 0.87 39.44±0.5239.44^± 0.52 53.50±0.6153.50^± 0.61 30.67±0.0930.67^± 0.09 0.670.67 0.740.74 Compression Ratio 0.75 Random 8.92±4.338.92^± 4.33 33.54±0.3433.54^± 0.34 50.60±1.0450.60^± 1.04 9.56±0.179.56^± 0.17 0.400.40 0.490.49 SpeechPrune 7.85±1.527.85^± 1.52 35.91±0.4835.91^± 0.48 51.30±0.3951.30^± 0.39 2.50±0.022.50^± 0.02 0.460.46 0.490.49 SparseVLM†‡ 8.19±2.088.19^± 2.08 38.94±0.4338.94^± 0.43 51.80±0.4851.80^± 0.48 13.97±0.0313.97^± 0.03 0.470.47 0.600.60 LTBM 6.89±2.346.89^± 2.34 38.04±0.8038.04^± 0.80 51.60±0.9151.60^± 0.91 10.80±0.0910.80^± 0.09 0.450.45 0.490.49 AudioTETRIS† 6.71±2.086.71^± 2.08 39.07±0.5239.07^± 0.52 52.50±0.3852.50^± 0.38 30.04±0.3230.04^± 0.32 0.450.45 0.640.64 † In-LM pruning (token elimination performed inside the LLM backbone). ‡ Originally developed for the vision modality, adapted here for speech/audio. Table 5: Extended results for Audio Flamingo 3 across all compression ratios. Method LibriSpeech WER ↓ IEMOCAP Acc. ↑ MMAU-mini Acc. ↑ CoVoST2 BLEU ↑ Latency (s) TFLOPs No Compression 3.61±0.543.61^± 0.54 59.11±0.8759.11^± 0.87 62.09±0.4562.09^± 0.45 37.02±0.7137.02^± 0.71 1.291.29 7.937.93 Compression Ratio 0.36 Random 4.02±1.214.02^± 1.21 58.43±0.6358.43^± 0.63 60.97±0.2360.97^± 0.23 36.82±0.1036.82^± 0.10 1.131.13 6.926.92 SpeechPrune 4.01±2.104.01^± 2.10 58.23±0.5958.23^± 0.59 61.42±0.8361.42^± 0.83 36.73±0.0536.73^± 0.05 1.151.15 7.097.09 SparseVLM†‡ 3.78±1.453.78^± 1.45 57.34±0.3257.34^± 0.32 58.83±0.6858.83^± 0.68 36.18±0.0636.18^± 0.06 1.141.14 7.317.31 LTBM 3.83±0.383.83^± 0.38 58.11±0.4158.11^± 0.41 61.00±0.6561.00^± 0.65 35.89±0.0235.89^± 0.02 1.211.21 6.986.98 AudioTETRIS† 3.71±0.453.71^± 0.45 58.94±0.5958.94^± 0.59 61.77±0.3761.77^± 0.37 37.31±0.1937.31^± 0.19 1.151.15 7.347.34 Compression Ratio 0.51 Random 6.67±0.916.67^± 0.91 56.45±0.6256.45^± 0.62 59.62±0.2859.62^± 0.28 34.72±0.1534.72^± 0.15 0.800.80 5.615.61 SpeechPrune 5.18±1.495.18^± 1.49 57.81±0.9757.81^± 0.97 61.19±0.2961.19^± 0.29 35.18±0.1635.18^± 0.16 0.870.87 5.715.71 SparseVLM†‡ 4.97±1.324.97^± 1.32 57.39±0.4957.39^± 0.49 58.14±0.5258.14^± 0.52 36.30±0.3236.30^± 0.32 0.920.92 6.186.18 LTBM 5.87±0.185.87^± 0.18 58.91±0.5658.91^± 0.56 60.79±0.2660.79^± 0.26 35.17±0.1335.17^± 0.13 0.910.91 5.635.63 AudioTETRIS† 4.62±0.974.62^± 0.97 58.85±0.3458.85^± 0.34 61.59±0.7261.59^± 0.72 36.26±0.0936.26^± 0.09 0.850.85 6.166.16 Compression Ratio 0.64 Random 10.91±1.0510.91^± 1.05 53.29±0.3353.29^± 0.33 58.19±0.5258.19^± 0.52 33.05±0.3133.05^± 0.31 0.730.73 4.534.53 SpeechPrune 6.59±1.926.59^± 1.92 56.71±0.6356.71^± 0.63 60.48±0.6460.48^± 0.64 34.87±0.2634.87^± 0.26 0.780.78 4.564.56 SparseVLM†‡ 6.08±1.946.08^± 1.94 53.98±0.6253.98^± 0.62 56.39±0.9356.39^± 0.93 35.39±0.0935.39^± 0.09 0.820.82 5.145.14 LTBM 6.49±0.716.49^± 0.71 57.18±0.2157.18^± 0.21 59.23±0.2059.23^± 0.20 33.90±0.0433.90^± 0.04 0.830.83 4.604.60 AudioTETRIS† 5.72±0.635.72^± 0.63 58.31±0.2658.31^± 0.26 59.85±0.1359.85^± 0.13 35.93±0.1535.93^± 0.15 0.770.77 5.145.14 Compression Ratio 0.75 Random 15.69±1.1215.69^± 1.12 51.89±1.0851.89^± 1.08 54.58±0.4254.58^± 0.42 27.72±0.2027.72^± 0.20 0.510.51 3.593.59 SpeechPrune 11.48±2.6311.48^± 2.63 54.78±0.5354.78^± 0.53 59.59±0.9759.59^± 0.97 31.98±0.2931.98^± 0.29 0.630.63 3.683.68 SparseVLM†‡ 9.59±1.279.59^± 1.27 52.60±0.4952.60^± 0.49 53.21±0.2853.21^± 0.28 34.25±0.0234.25^± 0.02 0.620.62 4.254.25 LTBM 8.94±0.338.94^± 0.33 57.31±0.2957.31^± 0.29 57.13±0.4057.13^± 0.40 33.01±0.0633.01^± 0.06 0.660.66 3.523.52 AudioTETRIS† 6.39±1.036.39^± 1.03 58.09±0.3758.09^± 0.37 60.92±0.2960.92^± 0.29 35.94±0.1435.94^± 0.14 0.600.60 4.264.26 † In-LM pruning (token elimination performed inside the LLM backbone). ‡ Originally developed for the vision modality, adapted here for speech/audio. A.3.2 Extended Component-wise Ablation Section 6.2 reports the component and stage ablation of AudioTETRIS’s fused score st(ℓ)=p~t(ℓ)⋅wts_t^( )= p_t^( )· w_t at compression ratio 0.510.51 only. Here we provide the extended results across all four evaluated compression ratios (0.360.36, 0.510.51, 0.640.64, 0.750.75) and all four benchmarks (LibriSpeech, IEMOCAP, MMAU-mini, CoVoST2), comparing the isolated single-stage variants (Only Stage 1, Only Stage 2) against the full two-stage AudioTETRIS pipeline (Stage 1 + Stage 2). Table 6 shows that combining both stages consistently outperforms either stage in isolation across every compression ratio and benchmark, with the gap widening substantially at higher compression - confirming that the phonetic transition prior and the depth-adaptive attention score contribute complementary, rather than redundant information. Table 6: Ablation results comparing individual stages against full AudioTETRIS across different compression ratios. The best-performing combination per compression ratio block is highlighted. Method LibriSpeech WER ↓ IEMOCAP Acc. ↑ MMAU-mini Acc. ↑ CoVoST2 BLEU ↑ Compression Ratio 0.36 Only Stage 1 5.50±0.255.50^± 0.25 57.45±0.3957.45^± 0.39 41.80±0.6241.80^± 0.62 31.21±0.0931.21^± 0.09 Only Stage 2 9.21±0.369.21^± 0.36 57.30±1.1957.30^± 1.19 41.70±0.8041.70^± 0.80 30.93±0.0430.93^± 0.04 Stage 1 + Stage 2 4.52±0.214.52^± 0.21 58.04±0.8458.04^± 0.84 42.36±0.2142.36^± 0.21 31.96±0.0431.96^± 0.04 Compression Ratio 0.51 Only Stage 1 7.39±0.097.39^± 0.09 56.37±1.5556.37^± 1.55 41.90±0.4241.90^± 0.42 31.18±0.0831.18^± 0.08 Only Stage 2 9.42±0.329.42^± 0.32 56.22±0.2156.22^± 0.21 41.20±0.8541.20^± 0.85 30.55±0.1330.55^± 0.13 Stage 1 + Stage 2 5.77±0.335.77^± 0.33 57.03±0.4157.03^± 0.41 42.91±0.3642.91^± 0.36 31.45±0.0631.45^± 0.06 Compression Ratio 0.64 Only Stage 1 8.92±0.468.92^± 0.46 56.68±1.4356.68^± 1.43 41.50±0.1441.50^± 0.14 30.11±0.0430.11^± 0.04 Only Stage 2 9.62±0.229.62^± 0.22 55.77±0.5755.77^± 0.57 38.70±1.1338.70^± 1.13 30.85±0.1930.85^± 0.19 Stage 1 + Stage 2 6.62±0.296.62^± 0.29 57.90±0.7657.90^± 0.76 41.83±0.1541.83^± 0.15 31.24±0.1131.24^± 0.11 Compression Ratio 0.75 Only Stage 1 13.78±0.2313.78^± 0.23 55.02±0.1455.02^± 0.14 40.80±0.4240.80^± 0.42 30.24±0.0930.24^± 0.09 Only Stage 2 15.56±0.5715.56^± 0.57 50.94±1.0650.94^± 1.06 36.10±0.5736.10^± 0.57 24.29±0.1124.29^± 0.11 Stage 1 + Stage 2 8.50±0.188.50^± 0.18 55.34±1.2555.34^± 1.25 41.46±0.3241.46^± 0.32 31.33±0.0731.33^± 0.07 A.3.3 Single-Stage and Alternative Multi-Stage Schedules Section 5.1 fixes AudioTETRIS’s schedule at two stages, layers 1,15\1,15\, uniform per-stage split, with layer 1 placed for compute savings (Appendix A.2.2) and layer 15 set by crossover calibration on Qwen2Audio (Appendix A.2.2). Here we ablate two orthogonal design choices: where in the decoder the pruning stages are placed and how the compression budget is allocated across stages once placement is fixed. Schedule parameterization. For stages at layers (ℓ1,…,ℓn)( _1,…, _n) and target keep-fraction K=1−rateK=1-rate, we assign each stage a weight wiw_i with ∑iwi=1 _iw_i=1 and set the per-stage retained fraction ri=Kwi,∏i=1nri=K,r_i=K^w_i, _i=1^nr_i=K, (9) so every variant reaches the same target compression rate regardless of how wiw_i is split. Equal weights wi=1/nw_i=1/n give the uniform split used by the main text; w1>w2w_1>w_2 front-loads the budget toward the earlier stage, and the reverse back-loads it. Dense variants keep n large and the split uniform, spreading small drops across many stages instead of a few large ones. Table 7 lists all variants tested. Table 8 reports the resulting per-stage compression rate (1−ri)×100%(1-r_i)× 100\% at each evaluated compression rate. For uniform-split variants this is a single number repeated n times; for front-/back-loaded it differs stage to stage, with front-loaded removing a larger share early and back-loaded the reverse. Table 7: Candidate pruning schedules. Placement variants use the uniform split; front-/back-loaded vary only wiw_i at the default layers 1,15\1,15\. Schedule Layers n stages Split Single-stage @ 1 1 1 – Single-stage @ 15 15 1 – Two-stage 1,15\1,15\ (default) 1, 15 2 Uniform Two-stage 1,19\1,19\ 1, 19 2 Uniform Three-stage 1,8,15\1,8,15\ 1, 8, 15 3 Uniform Front-loaded 1, 15 2 w=(0.7,0.3)w=(0.7,0.3) Back-loaded 1, 15 2 w=(0.3,0.7)w=(0.3,0.7) Dense (every ∼ 3 layers, full depth) 1, 4, 7, 10, 13, 16, 19, 22, 25, 28, 31 11 Uniform Table 8: Per-stage token-drop rate (% of currently live tokens removed at that stage) for each schedule, at every evaluated compression rate. Equal per-stage rates are written once as “X%X\% (×n× n)”; front-/back-loaded list rates stage by stage. Schedule Layers rate 0.36 rate 0.51 rate 0.64 rate 0.75 Single-stage @ 1 1 36% 51% 64% 75% Single-stage @ 15 15 36% 51% 64% 75% Two-stage (default) 1, 15 20% (×2) 30% (×2) 40% (×2) 50% (×2) Two-stage 1,19\1,19\ 1, 19 20% (×2) 30% (×2) 40% (×2) 50% (×2) Three-stage 1, 8, 15 14% (×3) 21% (×3) 29% (×3) 37% (×3) Front-loaded 1, 15 27%/13% 39%/19% 51%/26% 62%/34% Back-loaded 1, 15 13%/27% 19%/39% 26%/51% 34%/62% Dense (11 stages) 1,4,…,31 4% (×11) 6% (×11) 9% (×11) 12% (×11) Table 9: Schedule ablation, LibriSpeech WER (%, lower is better), Qwen2-Audio, across compression rates. Schedule rate 0.36 rate 0.51 rate 0.64 rate 0.75 Single-stage @ 1 6.51±0.146.51^± 0.14 8.92±0.178.92^± 0.17 12.40±0.3612.40^± 0.36 18.78±0.4118.78^± 0.41 Single-stage @ 15 5.44±0.615.44^± 0.61 6.93±0.426.93^± 0.42 8.94±0.488.94^± 0.48 12.89±0.8212.89^± 0.82 Two-stage 1,15\1,15\ (default) 4.52±0.214.52^± 0.21 5.77±0.335.77^± 0.33 6.62±0.296.62^± 0.29 8.50±0.188.50^± 0.18 Two-stage 1,19\1,19\ 4.78±0.384.78^± 0.38 6.02±0.356.02^± 0.35 6.94±0.626.94^± 0.62 8.96±0.318.96^± 0.31 Three-stage 1,8,15\1,8,15\ 4.36±0.714.36^± 0.71 5.93±0.715.93^± 0.71 6.47±0.726.47^± 0.72 8.12±0.738.12^± 0.73 Front-loaded 4.99±0.694.99^± 0.69 6.72±0.766.72^± 0.76 7.68±0.297.68^± 0.29 10.73±0.2810.73^± 0.28 Back-loaded 5.73±0.315.73^± 0.31 7.34±0.627.34^± 0.62 9.38±0.349.38^± 0.34 14.53±0.4214.53^± 0.42 Dense (11 stages) 7.13±0.457.13^± 0.45 8.79±0.578.79^± 0.57 10.09±0.6210.09^± 0.62 12.92±0.3612.92^± 0.36 Table 10: Efficiency and downstream-task trade-offs at compression rate 0.51. Latency is relative to the uncompressed baseline (1.00×1.00×). Schedule WER IEMOCAP MMAU CoVoST Latency TFLOPs Single-stage @ 1 8.92±0.178.92^± 0.17 53.55±0.5053.55^± 0.50 39.81±0.4139.81^± 0.41 27.05±0.0527.05^± 0.05 0.980.98 5.515.51 Single-stage @ 15 6.93±0.426.93^± 0.42 55.83±0.5255.83^± 0.52 41.52±0.2441.52^± 0.24 29.53±0.0929.53^± 0.09 0.990.99 5.925.92 Two-stage 1,15\1,15\ (default) 5.77±0.335.77^± 0.33 57.03±0.4157.03^± 0.41 42.91±0.3642.91^± 0.36 31.45±0.0631.45^± 0.06 1.001.00 6.136.13 Two-stage 1,19\1,19\ 6.02±0.356.02^± 0.35 57.41±0.4757.41^± 0.47 43.18±0.1843.18^± 0.18 31.69±0.0731.69^± 0.07 1.031.03 6.136.13 Three-stage 1,8,15\1,8,15\ 5.93±0.715.93^± 0.71 55.68±0.3755.68^± 0.37 41.67±0.7241.67^± 0.72 31.45±0.1031.45^± 0.10 1.041.04 6.326.32 Front-loaded 6.72±0.766.72^± 0.76 56.91±0.1956.91^± 0.19 41.48±0.3941.48^± 0.39 30.20±0.1830.20^± 0.18 1.021.02 5.715.71 Back-loaded 7.34±0.627.34^± 0.62 55.93±0.3155.93^± 0.31 41.44±0.5341.44^± 0.53 26.68±0.0426.68^± 0.04 1.061.06 6.216.21 Dense (11 stages) 8.79±0.578.79^± 0.57 57.62±0.8157.62^± 0.81 43.30±0.6443.30^± 0.64 30.90±0.0730.90^± 0.07 1.211.21 6.936.93 Results. We find that single-stage compression at layer 15 (6.93% WER at rate 0.51; Table 9) consistently outperforms layer 1 (8.92%), highlighting the role of text-conditioned attention in token ranking. Splitting the compression budget across two stages improves upon single-stage baselines across all rates, regardless of the chosen layers. Within two-stage setups, our default 1,15\1,15\ schedule achieves the strongest performance (5.77% at rate 0.51), outperforming 1,19\1,19\ (6.02%) by 0.2-0.5 percentage points across the sweep. The three-stage schedule 1,8,15\1,8,15\ yields small gains over the two-stage default at three of the four evaluated rates - 4.36%4.36\% vs. 4.52%4.52\% WER at rate 0.360.36, 6.47%6.47\% vs. 6.62%6.62\% at rate 0.640.64, and 8.12%8.12\% vs. 8.50%8.50\% at rate 0.750.75 - but this advantage is not monotonic: at rate 0.510.51 the ordering reverses, with the two-stage default outperforming three-stage (5.77%5.77\% vs. 5.93%5.93\%). Given this inconsistency and the added latency overhead of a third pruning stage (Table 10), we retain the two-stage 1,15\1,15\ schedule as the default configuration used throughout the main text, rather than the marginally stronger but less consistent three-stage variant. Redistributing the budget within two-stage setups reveals a clear asymmetry: front-loading (larger early cut) remains competitive with the default (4.99% vs. 4.52% at rate 0.36), whereas back-loading degrades sharply under higher compression (14.53% vs. 10.73% at rate 0.75). This aligns with our analysis in Appendix A.2.2: heavy pruning in later layers, where cross-modal attention dominates, incurs a higher penalty than pruning early while representations remain largely intra-audio. Finally, the 11-stage dense schedule underperforms at all rates, showing that frequent, fine-grained pruning could accumulate more errors for the ASR task. Table 10 highlights the efficiency trade-offs at rate 0.51. We observe that the two-stage compression schedule offers one of the most favorable balance between accuracy and computational cost. In contrast, three-stage and dense schedules introduce a slight overhead (1.04×1.04× and 1.21×1.21× latency, respectively) without consistent downstream gains, confirming the 2-stage compression is an effective overall configuration. A.3.4 Qualitative Examples of Token Survival Figure 12 shows token survival at the final pruning layer for three representative LibriSpeech utterances. The examples are consistent with our quantitative finding (§6) that the two stages contribute complementary rather than redundant information: Stage 1 supplies dense, distributed protection against losing phonetic detail, while Stage 2 contributes a smaller number of additional, task-relevant tokens that Stage 1 alone would not have prioritized. (a) Example 1: “their piety would be like their names, like their faces, like their clothes…a gift tenfold more acceptable than his elaborate adoration” (b) Example 2: “now you will see in these studies that the moment the white is inclosed properly…the little diamonds in the round window will tell as jewels if they are gradated justly” (c) Example 3: “well then, last, here is turner’s greek school of the highest class, and you define his art absolutely as first the displaying intensely and with the sternest intellect of natural form as it is, and then the envelopment of it with cloud and fire” Figure 12: Token survival at the final pruning layer for three representative utterances (ground-truth transcripts shown under each panel). Rows within each panel (top to bottom): mel spectrogram, Only Stage 1 (phoneme-transitions only), Only Stage 2 (attention-only), Stage 1 + Stage 2 (attention × phoneme-transitions, our full method). Orange = kept uniquely by Only Stage 1; mauve = kept uniquely by Only Stage 2; blue = kept uniquely by Stage 1 + Stage 2; grey = pruned.